what i read from the release note of L4T 24.1: further release are planned to support only 64-bit user space.
this means on L4T 24.2, 32-bit user space program cannot work, but this is very confusing for me. can you please clarify below questions for me? thanks.
if i compile a 32-bit statically linked program and execute it, it should work well, right?
if i compile a 32-bit program which has dynamic linked library, it cannot be executed because NV does not provide 32 bit library like glibc, right?
on L4T24.2, since CUDA will be 64-bit library, all 32-bit user space program which is linked with CUDA lib will not work, right?
In both R23.x and R24.1 the kernel is 64-bit. In R23.x user space is 32-bit. In R24.1 you have the option of a fully 64-bit user space, although a 32-bit sample rootfs is also available (making it a lot like R23.2). R24.2 should come out soon (it isn’t released yet) and is a 64-bit release expected to fix many issues.
About 32-bit and 64-bit on ARM platforms…
A typical desktop computer is x86_64, but can have i686 compatibility environment installed to support 32-bit x86 applications. The CPU supports this, and with user space also supporting this, 32-bit can run on x86_64.
An ARM Cortex-A57 is similar in that its main and native format is 64-bit. This is the ARMv8-a architecture and instruction set. The previous 32-bit generation is the ARMv7 instruction set. The Cortex-A57 also has a compatibility mode not unlike a desktop CPU…32-bit is also supported in a backwards compatible way, but there are some differences in the environment you’d probably need to know about.
The Cortex-A57 in 32-bit compatibility mode is the ARMv8 architecture (contrast this to 64-bit ARMv8-a). ARMv8 is able to use all older ARMv7 code. Compilers generating ARMv8 32-bit mostly put out ARMv7 code…but there can be some differences. ARMv7 will run on ARMv8, but you are not guaranteed that ARMv8 will run on ARMv7. Further, a hybrid ARM user space environment and kernel support are not as evolved as x86_64 with i686. The compiler in the desktop world supports both types of code in a single compiler…for ARM you need two separate compilers, two separate binutils, so on. This ARMv8 compatibility mode is why compiling a TX1 kernel requires two compilers…Linaro never merged compatibility code into the 64-bit compiler (there is a strong chance this will never be merged, or perhaps not for a very long time).
So if you want to run 32-bit applications, you can install R23.x or R24.1 with the 32-bit sample rootfs…in which case you won’t be using 64-bit. Or you can install R24.x with 64-bit support and run 64-bit applications…but you won’t be running 32-bit applications. Or you can install R24.x 64-bit and then add in 32-bit foreign architecture support to run either 32-bit or 64-bit…but this has not evolved and you may or may not have good luck depending on what you are doing. Even if you install 32-bit foreign architecture compatibility in R24.x you would be advised to do all 32-bit compiles via cross-compile from a separate host. Mixing 32-bit ARMv8 environment with 64-bit ARMv8-a is not a smooth process (yet).
when install the Jetpack, i chose the third option which is 64-bit. so on TX1 target, i found that only 64-bit library can be found, for example the libc and ld are all 64-bit and there is no 32-bit version there. this means i can only run 64 bit application on TX1 if it is linked dynamically, right?
i compiled a 32-bit application which has static link on my host ubuntu PC by below steps, but it failed to be executed:
1). installed gcc, the gcc -v showed below information:
2). write a simple hellow world program which only call printf("hello world") in main function, compiled by below comamnd:
[b] arm-linux-gnueabihf-gcc test.c -static -o test[/b][i][/i]
3). file test showed below information, it is 32-bit arm elf:
~/work/test$ file test
test: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=68f93da9083b3f97980a6e314204f7e7a9cf9394, not stripped
4). scp from host PC to TX1 and execute the program test. but it failed with below information:
[b]ubuntu@tegra-ubuntu:~/work$ ./test
Killed[/b][i][/i]
my program test is an elf file which is statically linked, so it does not need the ld file, also it will not try to find any library like libc, why it cannot be executed on TX1 24.1 64-bit environment?
Yes, dynamically linked 64-bit apps should work with this 64-bit linker environment. 32-bit dynamically linked apps would fail.
Compiling a 32-bit app non-natively is more involved than building a kernel. The kernel does not depend on outside software, but a user space application is connected to all kinds of support. In the case of cross compiles, kernel builds do not need cross binutils, but you do for anything else (this includes building the compiler itself…cross binutils may still be indirectly involved in kernel cross compile). Assuming this is done correctly you may also need a kernel feature to be able to operate in compatibility mode (ARMv8 32-bit is compatible with ARMv7/armhf). I believe this is enabled by default, but if you have done your own custom kernel compile you’d have to be sure the ARMV7 items are enabled.
If you have your executable on the Jetson you might see what Jetson thinks of the executable with command “ldd the_executable” (this will list outside resource requirements for linking). Hopefully ldd says nothing or validates static linking. Note that even if a program does not require linking to libraries via the linker that it does still need some bootstrapping external from the program. If you are programming bare metal you can’t use function “main” to start your C program…a statically linked executable is not necessarily independent of the environment which bootstraps up to function main…it just means that if you are able to start execution of main that from beyond main no outside support other than kernel system calls are required.
In order to reach main you will see files of the name format “crt#.*” (where ‘#’ is a number like 0 or 1) being required…these are architecture dependent. The 64-bit version is under “/usr/lib/aarch64-linux-gnu/”, the 32-bit version is most likely under “/usr/arm-linux-gnueabihf/lib/”, but there are some variations in placement…the part which is constant about these files is the architecture-dependent directory name somewhere in the parent directory. Try this:
sudo find / -name 'crt[0-9].*'
On 64-bit ARMv8-a the 32-bit armhf or ARMv8 is the foreign architecture. On a 64-bit desktop system i686 is a foreign architecture. During compile of your executable the correct crt file must be used or execution can’t succeed in the intended environment. At the moment of static linking and compile an incorrect crt file would make the executable non-functional by not correctly reaching function main during start up. In a cross tool user space build environment this would be something required to be available in the sysroot.
The version of crt which becomes part of your program depends on the machine it was compiled from. I’m not where I can check for what needs to be where right now, but one thing you can do when you execute the program is log the system calls to see which part fails. strace puts out a LOT of C-like logging which is basically all of the communications between program and kernel…many of those calls are expected to fail, e.g., it tries out each and every LOCALE and perhaps only en_US will not fail. The last 100 lines or so of an strace log may have a failure though which points to a missing file related to armhf/ARMv7/ARMv8-32-bit. You could log something like this:
strace -o trace_log.txt ./progname
Then to see only the last 100 lines “tail -n 100 trace_log.txt | less”. My guess is that something required for foreign architecture is missing (32-bit is not the native architecture and can be treated as foreign even though the CPU supports it).
Strace should have shown far more, perhaps the wrong crt file was used (each architecture has its own versions, but you’ll notice in Linux there is also crt0 and crt1…using the wrong one could be an issue before function main ever starts…I see you have only crt1).
If your test_32s program is compiled with debug symbols not stripped, then “nm” can possibly give some information about the binary. What is the output of:
nm test_32s | grep 'main'
Also, what is the output of:
objdump -a test_32s
NOTE: objdump may need a version specific to an archtecture if test_32s is a different architecture…run the command to find out.