Question about cross-compilation link errors

To cross-compile the l4t-multimedia included in Jetpack 5.0.2, we followed the guide on the URL( Jetson Linux API Reference: Setting Up Cross-Platform Support | NVIDIA Docs ), but a link error occurred.
/opt/nvidia/bootlin-toolchain-gcc-93/bin/…/lib/gcc/aarch64-buildroot-linux-gnu/9.3.0/…/…/…/…/aarch64-buildroot-linux-gnu/bin/ld: cannot find crt1.o: no such file or directory
/opt/nvidia/bootlin-toolchain-gcc-93/bin/…/lib/gcc/aarch64-buildroot-linux-gnu/9.3.0/…/…/…/…/aarch64-buildroot-linux-gnu/bin/ld: cannot find crti.o: no such file or directory

When I checked, the cross-compiler version was 9.3.0, and the compiler included in JetPack 5.0.2 was 9.4.0.
After compiling l4t-multimedia as a 9.3.0 cross-compiler, I think there is a problem referring to the library compiled with 9.4.0 in target_rootfs set to sysroot.

My guess may be wrong, but please check.
Also, please review whether it is not a problem that the tool chain and the Native GCC version are different.

Please reply.
Thank you.

The document does not mention setting up the linker path. When you compile, are you using a CMakefile or Makefile? Is there a linking setup in your file? If so, then it could be amended to add the path to those files.

Normally, on the Orin which was cloned (or from the mount point of the clone), do you see these files?

  • ls -l /usr/lib/aarch64-linux-gnu/crt1.o
  • ls -l /usr/lib/aarch64-linux-gnu/crti.o

If so, then likely your cross linker path just needs to add “/usr/lib/aarch64-linux-gnu” to the path (adjust for the mount point of the clone). If the files are missing, then you might need to install package “libc6-dev” (if you’ve already cloned, then it is simple to install to the original system, followed by an rsync update of the loopback mounted clone…just ask if you need to do this; or reclone after adding the dev package).

I’m using the MakeFile in 00_video_decode.
I’m using TARGET_ROOTFS as the mount path for that MakeFile, and the linker path is also set.
The file also exists in that path.

I followed the instructions, but the problem still occurs.

If you copy the corresponding crt1.o crti.o, crtn.o file to the path where the source code is located and build it, it will be solved, but I think it’s not the right way.

When VERBOSE is enabled and built, the path to crt1.o and crti.o appears to be set to the current directory during the collect2 process.

Below is the collect2 log.

  • /opt/nvidia/bootlin-toolchain-gcc-93/bin/…/libexec/gcc/aarch64-buildroot-linux-gnu/9.3.0/collect2 -plugin /opt/nvidia/bootlin-toolchain-gcc-93/bin/…/libexec/gcc/aarch64-buildroot-linux-gnu/9.3.0/liblto_plugin.so -plugin-opt=/opt/nvidia/bootlin-toolchain-gcc-93/bin/…/libexec/gcc/aarch64-buildroot-linux-gnu/9.3.0/lto-wrapper -plugin-opt=-fresolution=/tmp/cc8MBuJ1.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --sysroot=/home/user/jetson/rootfs --eh-frame-hdr -dynamic-linker /lib/ld-linux-aarch64.so.1 -X -EL -maarch64linux -o video_decode crt1.o crti.o /opt/nvidia/bootlin-toolchain-gcc-93/bin/…/lib/gcc/aarch64-buildroot-linux-gnu/9.3.0/crtbegin.o -L/home/user/jetson/rootfs//usr/local/cuda-11.4/lib64 -L/home/user/jetson/rootfs/lib -L/home/user/jetson/rootfs/lib/aarch64-linux-gnu -L/home/user/jetson/rootfs/usr/lib/aarch64-linux-gnu -L/home/user/jetson/rootfs/usr/lib/aarch64-linux-gnu/tegra -L/opt/nvidia/bootlin-toolchain-gcc-93/bin/…/lib/gcc/aarch64-buildroot-linux-gnu/9.3.0 -L/opt/nvidia/bootlin-toolchain-gcc-93/bin/…/lib/gcc -L/opt/nvidia/bootlin-toolchain-gcc-93/bin/…/lib/gcc/aarch64-buildroot-linux-gnu/9.3.0/…/…/…/…/aarch64-buildroot-linux-gnu/lib/…/lib64 -L/opt/nvidia/bootlin-toolchain-gcc-93/bin/…/lib/gcc/aarch64-buildroot-linux-gnu/9.3.0/…/…/…/…/aarch64-buildroot-linux-gnu/lib -L/home/user/jetson/rootfs/lib -L/home/user/jetson/rootfs/usr/lib video_decode_csvparser.o video_decode_main.o /home/user/jetson_multimedia_api/samples/common/classes/NvUtils.o /home/user/jetson_multimedia_api/samples/common/classes/NvElementProfiler.o /home/user/jetson_multimedia_api/samples/common/classes/NvElement.o /home/user/jetson_multimedia_api/samples/common/classes/NvApplicationProfiler.o /home/user/jetson_multimedia_api/samples/common/classes/NvVideoDecoder.o /home/user/jetson_multimedia_api/samples/common/classes/NvDrmRenderer.o /home/user/jetson_multimedia_api/samples/common/classes/NvJpegEncoder.o /home/user/jetson_multimedia_api/samples/common/classes/NvBuffer.o /home/user/jetson_multimedia_api/samples/common/classes/NvLogging.o /home/user/jetson_multimedia_api/samples/common/classes/NvEglRenderer.o /home/user/jetson_multimedia_api/samples/common/classes/NvBufSurface.o /home/user/jetson_multimedia_api/samples/common/classes/NvJpegDecoder.o /home/user/jetson_multimedia_api/samples/common/classes/NvVideoEncoder.o /home/user/jetson_multimedia_api/samples/common/classes/NvV4l2ElementPlane.o /home/user/jetson_multimedia_api/samples/common/classes/NvV4l2Element.o -rpath-link=/home/user/jetson/rootfs -rpath-link=/home/user/jetson/rootfs/lib/aarch64-linux-gnu -rpath-link=/home/user/jetson/rootfs/usr/lib/aarch64-linux-gnu -rpath-link=/home/user/jetson/rootfs/usr/lib/aarch64-linux-gnu/tegra -rpath-link=/home/user/jetson/rootfs//usr/local/cuda-11.4/lib64 -lpthread -lv4l2 -lEGL -lGLESv2 -lX11 -lnvbufsurface -lnvbufsurftransform -lnvjpeg -lnvosd -ldrm -lcuda -lcudart -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /opt/nvidia/bootlin-toolchain-gcc-93/bin/…/lib/gcc/aarch64-buildroot-linux-gnu/9.3.0/crtend.o crtn.o

I succeeded in the following way.

Create and build a symlink of crt1.o, crt1.o, crtn.o files in /usr/lib.
- Run the command below in the /usr/lib path
- ln -sf ./aarch64-linux-gnu/crti.o crti.o
- ln -sf ./aarch64-linux-gnu/crt1.o crt1.o
- ln -sf ./aarch64-linux-gnu/crtn.o crtn.o

The path referenced by the cross compiler does not match the path of rootfs on the board, which seems to be causing the problem.

Please check.

Hi jp.ko

You could refer to the following instructions for cross-compile on Jetpack 5.0.2

1. Clone the target rootfs on your Jetson board to your host system
$ sudo ./flash.sh -r -k APP -G clone.img jetson-agx-orin-devkit mmcblk0p1

2. Mount the `.raw` image with the following commands
$ cd $HOME
$ mkdir -p jetson
$ sudo mount -t ext4 clone.img.raw jetson
$ cd ~/jetson
$ sudo chown -R $USER:$USER .

3. Setup toolchain
Download: https://developer.nvidia.com/embedded/jetson-linux/bootlin-toolchain-gcc-93
$ mkdir $HOME/l4t-gcc
$ cd $HOME/l4t-gcc
$ tar xf <toolchain_archive>
$ export TARGET_ROOTFS=$HOME/jetson
$ export CROSS_COMPILE=$HOME/l4t-gcc/bin/aarch64-buildroot-linux-gnu-

4. Make the following symbolic links:
$ cd ${TARGET_ROOTFS}/usr/local
$ rm cuda
$ ln -sf cuda-11.4 cuda

$ cd ${TARGET_ROOTFS}/usr/lib/
$ ln -sf aarch64-linux-gnu/crti.o crti.o
$ ln -sf aarch64-linux-gnu/crt1.o crt1.o
$ ln -sf aarch64-linux-gnu/crtn.o crtn.o

$ cd ${TARGET_ROOTFS}/usr/lib/aarch64-linux-gnu
$ ln -sf libpthread.so.0 libpthread.so

5. Navigate to the directory for a sample and run `make` to cross-compile.
$ cd ${TARGET_ROOTFS}/usr/src/jetson_multimedia_api/samples/02_video_dec_cuda
$ make
1 Like

It’s the problem of path

Is the actual log of collect2 a single long line? If not, can you edit the post (the pencil icon in the lower right), and edit such that you put three backquotes (```) on a line by itself directly above the log, and again three backquotes on a line by itself below the log, or else highlight the log and click on the “code” icon (looks like “</>”)? This would preserve formatting. Only useful of course if the actual original log has formatting (wouldn’t hurt to do this even if the original log did not have formatting).

The URL you mentioned depends on setting up the loopback partition being mounted at:
$HOME/jetson/

Is that where it is mounted?

That Makefile inherits from the parent directory “Rules.mk” if you run from any parent. This block of that file sets up linker specs:

# All common dependent libraries
LDFLAGS += \
        -lpthread -lv4l2 -lEGL -lGLESv2 -lX11 \
        -lnvbuf_utils -lnvjpeg -lnvosd -ldrm \
        -lcuda -lcudart \
        -lnvinfer -lnvparsers \
        -L"$(TARGET_ROOTFS)/$(CUDA_PATH)/lib64" \
        -L"$(TARGET_ROOTFS)/usr/lib/$(TEGRA_ARMABI)" \
        -L"$(TARGET_ROOTFS)/usr/lib/$(TEGRA_ARMABI)/tegra"

Note that earlier in that file it should specify:

ifeq ($(TEGRA_ARMABI),)
TEGRA_ARMABI ?= aarch64-linux-gnu
endif

What that does is first test to see if environment variable “TEGRA_ARMABI” is empty; if it is, then it sets it to “aarch64-linux-gnu”. This would cause search to include the location with the “crt*.o” files:
-L"$(TARGET_ROOTFS)/usr/lib/$(TEGRA_ARMABI)"

…which expands to:
~/jetson/usr/lib/aarch64-linux-gnu
(and if that has crt*.o files, then linking should work)

However, it will fail if any of these occur:

  • The mount point differed for the clone.
  • If you cannot find the file from ls -l ~/jetson/usr/lib/aarch64-linux-gnu/crt*.o.
  • If environment variable “$TEGRA_ARMABI” already exists, and is not “aarch64-linux-gnu”.

If you are at the compile location, what do you see from “echo $TEGRA_ARMABI”? It should be empty.

If you run command “ls -l ~/jetson/usr/lib/aarch64-linux-gnu/crt*.o”, do you see various files named “crt*.o”? You should see them. If not, then on the actual Jetson the clone came from, do you see anything from:
ls -l /usr/lib/aarch64-linux-gnu/crt*.o

Hi KevinFFF.

It seems like the same guide as the way I succeeded.

Where is this guide included in the official document?

I could find a part of the following URL (Jetson Linux API Reference: Setting Up Cross-Platform Support | NVIDIA Docs), but I couldn’t find anything related to symlink.

I would appreciate it if the information was updated in the LT4 API document.

Thank you.

This updated cross-compile guide for Jetpack 5 would be included in the next released document.