How can I cross compile C/C++ code for the Jetson Nano?

[crose72]

Hi, thank you in advance for your time.

I would like to cross compile c/c++ code for my nano to run, but the resources on this topic are scarce and confusing to say the least. I’m trying to establish a workflow where I write code on my laptop, compile the code on my laptop, connect my nano to my laptop via micro USB, and then flash the executable to the nano from my laptop via USB. Then when I unplug the nano from my laptop and power it from the wall I expect my executable to run and control LEDs or my camera or whatever I’ve programmed it to do. This is how I work with my STM32 boards and arduino and I know it’s possible with the Jetson Nano. How can I do this? Additionally the ability to debug while connected to my laptop would be nice, if anyone knows how I can do that.

I’ve searched high and low and everything I find on this topic isn’t very helpful.

[linuxdev]

This isn’t a complete answer, but will be a place to start. This is more a couple of terms than it is procedure.

The first thing you need to know is that your desktop PC uses a different CPU architecture. What you’re interested in cross compiling is called either aarch64 or arm64 in the tools. The CPU would also be called ARMv8-a.

If you have the cross compiler, then building a kernel is the simplest build. Kernel space is the simplest because it is “bare metal”, and does not link to other library content. Regular programs are “user space”, and they get more complicated because they usually involve libraries. If there is a library, then you must also link, and cross linking complicates things.

So if it is the kernel, then you set up the environment to tell it you are cross compiling, and you are more or less ready to develop. If you are building a user space application, then you also need to set up the cross linker, which in turn means you need to have cross architecture libraries not present on the PC, and you also need to set up how that cross linker searches for libraries.

You can usually install cross compilers and cross linkers with apt (or for kernel tools, use the ones NVIDIA mentions for a given release). The libraries are the hard part. Although several tool vendors (e.g., Linaro) have some minimal library support, there’s often so much that you have to compile yourself that it can get complicated. On the other hand, if you have an eMMC model Jetson, and clone it, or an SD card model, and copy the libraries (the entire tree), you’ll have what you need if you install libraries on the Jetson (which you’d need anyway).

Linaro.org tends to supply everything you might need, although I again suggest for libraries you install with a clone of the Jetson instead of building up libraries it is a lot easier.

One would normally copy the program to the Jetson with “scp” (the “ssh” enabled version of remote copy).

As long as your PC is running its GUI you can do remote debugging over ssh (with port forwarding) using gdb (the cross version isn’t needed as long as gdb runs on the Jetson and only forwards to the PC). I don’t use eclipse most of the time, but if you want an integrated environment which can run on the PC and do remote copy and debugging in a more integrated way, then that is what you would most likely be using.

[crose72]

Okay so I believe I understand the first part about choosing aarch64 or arm64 as far as compilers go (I wanna say it’s the first from what I’ve read).

I’ve tried to put in my due diligence before replying, and I’ve spent countless hours trying things, reading documentation, all to try and get my development workflow set up.

I appreciate your explanation, but I’m still having a hard time putting together all of the information I find in order to actually compile and then flash code on the nano. I’m even trying to follow the instructions from NVIDIA themselves in the “Jetson Linux API Reference” which says to first install JetPack on the target (in my case a Jetson Nano) and then setting up the cross-compiler on the host, but when I follow the directions here

there is no JetPack on that page to even download, so I can’t really complete the rest of the instructions, which is frustrating. Everything on that site is for the Xavier, or AGX, or everything but the nano. And when I run a particular command it says I have JetPack installed, but then I still have to go to the downloads page from NVIDIA to download JetPack? Tell me that’s not confusing.

Also, I’ve done the “Getting Started with Jetson Nano” and put the image on my micro SD card, just fyi.

I am really stuck trying to get started with code development on this development kit and I’ve looked at well over a dozen different “guides” or resources from NVIDIA and others, and the general consensus seems to be that it’s confusing.

[linuxdev]

When you write a program, you don’t flash it. It’s just a file to copy from one computer to another. Flashing is reserved for installing an operating system when there is no operating system. ssh has the command “scp” to copy from one machine to another (there are other possibilities, but this is simple and reliable and fast).

So far as cross compiling CUDA code, and related, someone else will need to answer. However, there is something related to this which is probably confusing: A GPU also has an architecture. For NVIDIA GPUs there is something called “compute capability”. One can compile a CUDA program such that it applies to one or more GPUs, but it is neither ARM (arm64/aarch64) nor Intel (amd64/x86_64). For a Nano, which is Maxwell architecture. GPU code is compiled in general by the nvcc app, and options during compile specify which compute capability to add (and more than one can be specified, e.g., for the Jetson plus one for host PC). For the Nano it is compute capability 5.3, abbreviated as “sm_53”. Example:
nvcc -gencode arch=compute_53,code=sm_53 ...other content...
(you can have a list of architectures to support, the above is just for a Nano because it is Maxwell architecture)

Here is an example Makefile:
https://forums.developer.nvidia.com/t/tx2-gpu-obsolete-for-pytorch/158330/8
(note how it enables multiple architectures)

Basically, if you run a user space program, then the part which runs on the CPU must be for that CPU’s architecture, and the same is true for GPU: It is a specialty CPU with its own architecture, and one program can talk to another, or interchange data.

If you want to find software which matches with your Nano, then look first at the L4T release on the Nano with “head -n 1 /etc/nv_tegra_release”. Then go to the listing of L4T releases here:
https://developer.nvidia.com/linux-tegra

If your release is R32.7.2 (the latest release a Nano works with), then you’d end up here:
https://developer.nvidia.com/embedded/linux-tegra-r3272

Documentation and downloads will work with that release, including the example code which JetPack/SDK Mananger can install on the host PC, along with nvcc. Note that if you run another release of SDKM, then you can start it like this to see all available releases compatible with SDKM:
sdkmanager --archivedversions

The R32.7.2 “L4T Driver Package (BSP) Sources” has many files in it (you can use command line to view this, or the Ubuntu “ark” program). One such package is “nvsample_cudaprocess_src.tbz2”. Or you can use sdkmanager to directly just install examples on the host PC (you can uncheck installation of anything to the Jetson and just pick for host PC, but if you choose to also install to the Jetson, then you can be sure a number of important items are present for developing on the Jetson and cross compiling on the host PC).

So you have some extra challenges there: Cross compiling CPU code for the Jetson, cross linking to the Jetson (preferably linking to a local PC’s mounted clone of the Jetson), and building the GPU code to the right compute capability.

Someone else can probably give you a better way to start on this than I can.

[crose72]

So I have tried to set up the development environment according to NVIDIA API Building and Running, but it turns out that they left out the part where if you followed the instructions in Getting Started With Jetson Nano Developer Kit, then you don’t have to follow the instructions under “Building and Running.”

Then I tried to follow the instructions for NVIDIA API Setting Up Cross-Platform Support, and what they don’t tell you is that those instructions are correct, but incomplete. I had to go down the rabbit hole with Alice in wonderland and have tea with the Mad Hatter before I found out those instructions won’t work for my situation.

Basically, those instructions were incomplete (had to search for the intermediate steps all over the internet). The takeaway there was that using WSL2 won’t work for the cross compile environment because my laptop is not a true Linux metal OS (hardware wise), and so when connecting the nano to my laptop via usb, my laptop will not be able to pick up the usb connection from the nano (it’ll bounce back and forth between the WSL and windows).

I have since turned towards using containers for developing on my laptop and deploying onto the nano. I have had success getting a “Hello world” program to run on both my lapatop and the nano using the same container. I’m planning to move forward with this workflow.

Though I would be interested to know if I’m wrong about needing a Linux hardware OS (I know I messed that up to, hopefully the point comes across) and that I can use WSL.

[linuxdev]

I was recently told there is a workaround for this, but at least by default, WSL2 does not have a full Linux kernel, and part of what one must do during boot is use “loopback” mounting of files to make them pretend to be actual block storage devices (partitions). Every VM has the possibility of failing USB, although if you know that particular VM, then there should be a way to fix that (it just isn’t supported by NVIDIA since every VM is responsible for setting up USB correctly). Not having loopback though is a deal breaker, and it is a feature of the Linux kernel drivers, not some user space. I’m not sure of the process or difficulty, but someone not long ago said there is a way to add loopback capability to WSL2 (it’d be interesting if someone could reply how that is done).