32-bit or armv7 gcc toolchain for TX2?

Hi,

I have some code that ran on a TX1, that gets an error when running on the TX2.

I believe the error stems from a DLL/.so that I’m using which I think was built for 32bit ELF. When I run the built project, I get:

./<library_name>.so: wrong ELF class: ELFCLASS32

So I believe if I rebuild the project to target 32bit or armv7, I have a chance that the DLL will work fine. I check my toolchain, and I see this as some of the output info:
Target: aarch64-linux-gnu

So it targets 64bit by default (which it should, as this is a TX2). Instead I try and compile by giving it a -march argument such as

g++ -march=armv7 main.cpp

which outputs

main.cpp:1:0: error: unknown value ‘armv7’ for -march

So I am assuming my toolchain does not have armv7 included? Is this correct?

(I suppose I could also consider cross-compiling, but I am working directly on the TX2 at the moment)

Thanks for your time

a.panella.aus,

Maybe you can get some idea by referring to the cross-compile for kernel image.

https://developer.nvidia.com/embedded/dlc/l4t-documentation-28-2-ga
→ Kernel customization
→ Build NVIDIA kernel

Just FYI, only the very earliest releases of L4T on a TX1 (around R24.1) were 32-bit user space. If you used a more recent release (and almost every ancient release is newer than the 32-bit TX1 releases), then you wouldn’t have the issue.

If you add 32-bit applications running in a 64-bit environment, then performance will drop dramatically. You will also have to install all of the 32-bit compatibility linkers and libraries on the TX2…it gets invasive. If you were to rebuild everything from source on the TX2, then you might also get around this…just don’t link against armhf/arm32. You will save yourself a lot of grief if your TX1 is flashed to a 64-bit version (assuming you need TX1 and TX2 to use the same apps).

@linuxdev

I essentially inherited a project configured for TX1 with working code. Moved everything to TX2 and it doesn’t run due to .so being ELFCLASS32. So I thought if I recompiled targetting Armv7, I might be able to get around this. I am not certain what version of L4T was running on the TX1, but it probably is fairly old by now.

I think if I was to flash the TX1 to a 64bit version, the .so/DLL I have will not work, as I suspect it was built for 32 bit. When trying to build and run the same project on the TX2, I get errors to do with the .so and wrong ELF class (ELFCLASS32). Ideally, I would like to rebuild everything from source, but unfortunately I do not have access to the DLL source. I do think that this would be my best option though, do you agree that the best way is to get the dll source and rebuild it for the TX2 architecture?

Thanks for your time

Arm32, on both a TX1 and a TX2, is a compatibility mode. This is considered a “foreign architecture”. Now if you look at a desktop PC you will see it is native mode x86_64, but has the possibility of compatibility 32-bit packages (e…g, “i386” or “i686”). The compatibility packages are considered a foreign architecture even on a PC, but lots of packages exist for this and so all of the infrastructure exists. The linker, all libraries, and everything which is 32-bit must be added separately, and on a Jetson, this does not pre-exist. You must build or hunt around and take great pain to do so.

When the TX1 first came out it used 64-bit kernel space, but 32-bit user space. This is because aarch64/arm64 was brand new, and all of the software in the world which was ported to this came from armhf of the TK1. It was approximately (not sure if I remember correctly) L4T R24.2 release which transitioned from 32-bit user space to 64-bit user space. Prior releases had only the 32-bit compatibility linker and libraries. Thus, it was easy to use 32-bit…that user space was about to be ported to newer 64-bit though.

The TX1 is currently…and for a long time…64-bit user space. Even a newer release of L4T on a TX1 would consider the 32-bit as incompatible unless you were to build up an entire infrastructure to support this foreign architecture. R24.1 had only foreign architecture and did not support native architecture (well, kernel drivers were 64-bit, but all else outside of kernel space was not).

You have inherited a project built on a system which is many years out of date (see “head -n 1 /etc/nv_tegra_release” for the release version on any given Jetson). You have the right idea, but the reverse of what will keep you sane…migrate the TX1 instead of the TX2. Probably the biggest issue will be that the CUDA version will have changed.

Before considering any approach you might want to clone any root file system you need a reference to. You cannot use a clone from a different release to flash with, but you could always reinstall via clone to the same release if things go bad. You can also use a loopback mount of the clone to copy any content at any later date even if the Jetson is erased. Ask more if you need clone information, but beware that the answer changes depending on release (you’d need to report the “head -n 1 /etc/nv_tegra_release”).

You will need the source of your libraries if they are your libraries, and rebuild them on arm64. If they are linked against existing libraries, then it is probably just a case of compiling directly on the TX2 (or even TX1 if using a 64-bit release) to get correct linkage from existing libraries.

So a big question list is:

  • Which CUDA release might be used if using CUDA?
  • Are the libraries being linked against yours (custom), or are they something part of the operating system?
  • Which software, of that failing, is from binary software which you have no access to source code and which is not part of the operating system?

Understood, sounds like I should avoid this if at all possible.

Okay, I think you’re correct, I checked the TX1 and it is running L4T 23 release rev 1.1, EABI:hard. Some of the API files and DLL I have are contained in a folder named L4T21.3-JetsonTK1-4.5, so I’m guessing it is built for 32bit userspace, hence the ELFCLASS32 error. So I guess recompiling from source with a 64 bit target will likely fix my issues.

Alright, sounds like the time difference between L$T 23 and L4T 28.1 (What the TX2 is running) is huge.

When you say migrate the TX1, do you mean that I should update L4T? The idea is that I get legacy code from TX1 running on the TX2 for better performance. I am hesitant to risk modifying the TX1 in any way if the project will still be running on TX1 hardware anyway

This is the big question. The library being linked against is a DLL that I do not have the source for, and was developed as part of a product from a third party company. They design interface boards and provide an API via the DLL to communicate with the interface boards. This is why I do not have the source - it is their IP. By the way, if their API is doing USB3 interfacing, would this mean I would likely have to port their library to 64bit? Or do you think I can just recompile targeting 64 bit architecture and it will work?

Thanks for the detailed response, I do not have a lot of experience with arm or embedded nvidia boards.

You can be guaranteed that 100% of everything which was from a TK1 is 32-bit armhf. Anything CUDA on a TK1 is CUDA version 6.5 or earlier…no CUDA version was ever released beyond version 6.5 for 32-bit. Barring a need to update code to go from CUDA 6.5 to some newer CUDA (newer platforms don’t use the older CUDA), then recompiling should basically just make many parts “just work” (I won’t guarantee it, I have no idea of what the code actually links to or depends on).

You can clone the original root file system for later reference, but you couldn’t simply use it to flash the next system.

The meaning of “migrate the TX1” is to upgrade the version of L4T on the Jetson itself…to flash. The TX1 with 32-bit user space won’t do much good (this has been known to cause insanity! :P). You could stick to slightly older 64-bit releases, e.g., L4T R28.1. To do so though, you would also have to migrate any of your code to match this. If there is a dependency on CUDA, then this would be both the most work and the most meaningful gain against future pain by porting this (meaning by adjusting any older CUDA calls to the newer release). For example, you might have to port CUDA 6.5 code to CUDA 9…this would be more than just a recompile. Much of the migration would be a simple recompile, but not all would be a simple recompile.

About libraries: Most of what your applications will link to are part of the operating system. When you recompile, and if the libraries are part of the operating system, then you won’t have any “significant” (YMMV) work to deal with it…the compiler will simply link against the 64-bit version. In some cases (where for example new features are added and old features are altered) you may need to make changes. “Many” (speculation) of the libraries which you did not write and provide will “just work”. Many people have complicated builds and never create their own libraries, in which case you don’t need to port an actual library.

On an Ubuntu system, the “dpkg -S ” command can tell you what package a file comes from. On any Linux flavor, the “ldd ” command can tell you what libraries an executable is linked to. Combining ldd and dpkg commands you can get an idea of what dependencies are fulfilled by the operating system, versus which ones are custom requirements.

On a Jetson TX2, running R28.2 (this is an example…the commands will not differ with a TX1 on R28.1…this investigates “/bin/ls” only as an example), I can run “which ls” and see “/bin/ls”. If you see a symbolic link or alias, then you’d have to adjust for this, but ldd can then tell me what “ls” is linked to (FYI, I am in a root shell, “sudo -s” for all of this…in some cases this matters, in other cases it does not matter):

# ldd /bin/ls
        linux-vdso.so.1 =>  (0x0000007fa2b92000)
        libselinux.so.1 => /lib/aarch64-linux-gnu/libselinux.so.1 (0x0000007fa2b33000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007fa29ec000)
        /lib/ld-linux-aarch64.so.1 (0x0000005566599000)
        libpcre.so.3 => /lib/aarch64-linux-gnu/libpcre.so.3 (0x0000007fa297b000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007fa2967000)                                                                                                              
        <b>libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0</b> (0x0000007fa293b000)

If I were to run “ls -l” against these, it might be a symbolic link…then I’d run “ls -l” again against the provider of that symbolic link until I see the hard file (if not familiar, a symbolic link just points at another file and stands in place for the other file…“ls -l” uses “=>” to name the file the link points to similar to dereferencing a C pointer). As an example, I see “libpthread”, and this points to “/lib/aarch64-linux-gnu/libpthread.so.0”. Using “ls -l /lib/aarch64-linux-gnu/libpthread.so.0”, on this release, it points to hard link “/lib/aarch64-linux-gnu/libpthread-2.23.so” (a symbolic link is a pointer, a hard link is the final actual file). Now I can use the package tool (“dpkg”) to tell me what package provided the file:

# dpkg -S /lib/aarch64-linux-gnu/libpthread-2.23.so
<b>libc6</b>:<b>arm64</b>: /lib/aarch64-linux-gnu/libpthread-2.23.so

Because I know a system package provided libpthread (via the arm64 version of the “libc6” package), I know I won’t need to recompile this library when I migrate…instead I’ll just rebuild my app and the compile itself will take care of linking. Should it turn out that the application is linked to something not provided by the system, then I would have to build the library myself before I could build my application.

If a library is provided by someone else and you do not have the source code, then if they also provide an arm64/aarch64 version, you do not need to worry about it…this too will simply be linked against when you recompile. If it turns out the provider of a proprietary binary library does not have a version for arm64/aarch64, then you must ask the provider of that binary to port it and provide it to you. You won’t have any other way of dealing with it other than sticking to the old 32-bit version.

FYI, most packages have a name, and a “name-dev”. The “-dev” version is the one used for development, e.g., header files. So if you were to port something which links to libc6, then you might (or might not) find you need to install “libc6-dev”. Sometimes you will need to set up the apt tool’s configuration to show the dev version, but you can ask about that when you get there. If the repositories are configured already, then the dev version of libc6 would be as simple as:

sudo apt update
sudo apt-get install libc6-dev

The list would be large, but you can search for “libc6” via:

apt search libc6
# Or more narrow search:
apt search libc6-dev

@linuxdev

Thanks for all of this information, it is immensely helpful. I think this is the real part that determines if I will be able to port things to the TX2:

I am trying to get a hold of the source. My plan is to rebuild the .so from source targeting the TX2 architecture, and hope that it works. I guess there is a chance that it works as kernelspace is 64bit for both. If I had to port it myself, do you have any advice for that?

Otherwise, I can just continue to work on the TX1, which is not ideal, but it will work at least…

Often any kind of “./configure” step will allow you to simply name “arm64” or “aarch64” architecture and it will “just work” (or when compiled natively, then this will be auto detected and not even require a special config). There are cases where code is architecture dependent though, e.g., it uses assembler, and you’ll need significant knowledge. More likely the “./configure” step will allow you a chance to change features, and after any build error you’ll just adjust the “./configure” and get past that error…and keep going that way until it builds. You’ll probably need specific issues to quote from the actual build before more can be answered (e.g., it might use cmake instead of configure scripts).

Above all, I would suggest looking at what provides the package on another Linux system using its package tool. It isn’t unusual for people to get caught up trying to build from the manufacturer’s site when it turns out the tool or lib already exists by some other name…in which case a single command could have taken care of everything. If you do run into trouble with this library give specific details of what library it is and where it came from originally (if you can’t just add a package, then someone else will need to reproduce your build to get the error and give advice).

@linuxdev

Yes, my plan is to build it natively on the TX2 with the source and see if it works - my hope is that the source has no assemlber and is just C/C++.

Which package are you referring to here? The .so I am attempting to use will definitely not be provided as a repo on a linux distro, it is provided prebuilt only after the interface hardware is purchased. I suspect if the manufacturer provides me with the source, I will not be able to share it for IP protection purposes. If the rebuild doesn’t work natively targeting aarch64, then I suppose I will just use the TX1 and worry about the upgrade to TX2 later.

Thanks for your help @linuxdev, I believe you’ve provided me with enough information to proceed. I guess this didn’t end up being about 32 bit toolchains at all, but your most recent answer is good, I think I will accept that and reopen a fresh thread that is more relevant when I need further assistance. I appreciate your explanations, I learned a lot.

Well…the “which package” is actually “the question”…I am suggesting that if you have files, then there are cases where people think they must be acquired from a third party…but at times those files may already exist as part of a standard package. Since you know this is only available with purchased hardware it is likely that you are correct and the library is only available from a third party. On the other hand, even proprietary hardware often has a known chipset, and it is the chipset for which the driver must exist. An example would be that there are many Realtek chipset network cards provided by different manufacturers…all would likely use the same driver even if some part of the NIC is proprietary.

Any information you are allowed to share, or whatever you find to be an issue while trying to make the software work is useful.

@linuxdev

Sorry, now I understand what you are saying.

The package is interfacing to an FPGA which implements the USB3.0 spec. The FPGA is probably running a custom kind of USB3 which could be implementing unknown handshaking etc. I do not think it will be possible for me to find standard packages that perform the same function - in fact, the spec is probably unique for the sole purpose of encouraging a purchase.

Either way, going forward is dependent on manufacturer providing me with source, otherwise I get to enjoy developing for L4T R23 :(

Yes, this is probably a custom USB device and not a standard class. So you are correct this would require the manufacturer’s help.

Yes, I thought so. While we were corresponding, the manufacturer has confirmed they can port it for a fee. I am happy to let this happen as it saves me a lot of time porting it myself - Thanks for the recommendations and advice throughout the thread, I am sure this thread will help someone else in a similar position