qemu-system-arm...Using with system.img of flash.sh?

Robotics & Edge Computing Jetson & Embedded Systems Jetson TK1
1

I’ve successfully been cross compiling on the Jetson’s system.img created by flashing L4T R19.3. This is very promising as a full cross-compile environment, but lacks the ability to update packages from an x86 host. Thus, I’m trying to run the system.img under qemu, and possibly have apt-get and dpkg available for system.img update (flash.sh can then re-use this image…meaning faster flash times, and custom edits right from the start).

I’ve not used qemu before, and although qemu supports cortex-a15, Jetson itself is not a listed board. Can anyone here suggest the proper command line to qemu-system-arm to use system.img?

2

I’ve been using qemu-user for what you’re trying to do.

I have a Gentoo ARM rootfs at /var/chroot/armv7a, the script below as /var/chroot/enterarmv7a.sh.
And most importantly a qemu-arm binary for the native host (in my case, x86_64) in both /usr/local/bin
AND /var/chroot/armv7a/usr/local/bin.

now chdir into /var/chroot/armv7a and execute the enterarmv7a.sh script one level above:

armv7a # ../enterarmv7a.sh
>>>
>>> Configuring ARMv7-A environment
>>>
/var/chroot/armv7a
>>>  mounting filesystems
mount.nfs: Failed to resolve server moya: Name or service not known
enabled
enabled
interpreter /usr/local/bin/qemu-arm
flags: 
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff
>>> entering ARMv7-A environment at /var/chroot/armv7a
<< ARMv7A >> dargo / # uname -a
Linux dargo 3.16.0-rc7-00114-g8d71844-dirty #1 SMP PREEMPT Sun Aug 3 21:08:32 CEST 2014 armv7l GNU/Linux

note the “armv7l” arch!

This setup works well to compile all kinds of stuff “natively” with qemu.

the script:

#!/bin/bash
echo ">>>"
echo ">>> Configuring ARMv7-A environment"
echo ">>>"
echo $PWD
echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/local/b
echo ">>>  mounting filesystems"
mount -t proc proc $PWD/proc || echo "FAILED /proc"
mount -t sysfs sysfs $PWD/sys || echo "FAILED /sys"
mount --bind /dev $PWD/dev || echo "FAILED /dev"   
mount -t tmpfs -o rw,nosuid,nodev,noexec,relatime shm $PWD/dev/shm || echo "FAILED /dev/shm"

cat /proc/sys/fs/binfmt_misc/status
cat /proc/sys/fs/binfmt_misc/arm   
cp /etc/resolv.conf $PWD/etc/resolv.conf || echo "FAILED resolv.conf"
echo ">>> entering ARMv7-A environment at $PWD"

chroot $PWD bin/bash --login

echo "<<<"
echo "<<< Back to native environment!"
echo "<<<"
umount $PWD/dev/shm
umount $PWD/dev
umount $PWD/sys    
umount $PWD/proc
echo -1 > /proc/sys/fs/binfmt_misc/arm
3

Thanks for the reply…I guess I got closer, but I think differences on my host (fedora 19) may have caused failure. FYI, I edited for my mount point at /mnt/L4T_R19.3, which is loopback mounted from the Jetson’s system.img (which is completely updated with all of my customizations, including /etc/resolv.conf).

The first of two errors is that there is no:
/proc/sys/fs/binfmt_misc/arm

The /proc/sys/fs/binfmt_misc/status exists and is enabled. I don’t know how important that is, but the mount commands prior to this had succeeded (or at least the script did not complain), so if those result in the /proc files, it may be that my host kernel needs something added, or else fedora 19 is just different.

The second error could possibly just be the security setup of fedora 19, as chroot fails. As root, I have never succeeded in using chroot on this particular distribution. The error is (for all chroot):
chroot: failed to run command ‘/bin/bash’: Permission denied

I am wondering though, where is the qemu-system-arm called in? It seems like the script is setup in preparation, but lacks the qemu command.

4

oh line 6 of the script is incomplete:

echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/local/bin/qemu-arm:' > /proc/sys/fs/binfmt_misc/register

when this command succeeds, you get the /proc/sys/fs/binfmt_misc/arm entry.
the chroot fails because the kernel doesn’t know how to execute an ARM binary on an x64 host,
which the above line takes care of. It just tells the kernel to run ARM elfs with /usr/local/bin/qemu-arm.
That’s also why qemu-arm has to be inside the ARM rootfs, because the kernel looks for it
at the base of the chroot.

5

This works further than before, but it still seems like something to do with chroot on fedora 19 is different or has some security issue even run as root. It changes to the proper mount point of the chroot environment, and I have the qemu binaries all available at the point it looks; then fails at chroot:

~/testing/enterarmv7a.sh 
>>>
>>> Configuring ARMv7-A environment
>>>
/mnt/L4T_R19.3
>>> mounting filesystems
enabled
enabled
interpreter /usr/bin/qemu-arm
flags: 
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff
>>> entering ARMv7-A environment at /mnt/L4T_R19.3
chroot: failed to run command ‘bin/bash’: Permission denied
<<<
<<< Back to native environment!
<<<

The qemu-system-arm in both /usr/bin/ and in /mnt/L4T_R19.3/usr/bin/ are the same binary. Since it was executed as root, and since I can’t seem to find any chroot which works (always permission denied), I wonder if it is an SElinux issue or something else. Nothing is ever logged in any log file concerning chroot.

6

do you have the fs loop mounted? make sure the fs doesn’t have “noexec” set, and you run everything
as root.

7

This was the first thing I checked. I specifically added exec to be sure. Other x86 executables I add here can execute, but no chroot command is ever allowed under any circumstance.

It really seems like something in the fedora configuration is denying chroot…I can even copy to non-loopback mounted locations (all files needed for an app such as a simple hello_world) and chroot is denied permission. There must be some security issue under fedora, but since no log messages occur, it seems impossible to figure out. Strace just shows permission denied as well.

8

Sorry to revive an old post, I’ve followed the instructions in your post but I’m running into a bit of an issue. My host environment is ubuntu 14.04 x86_64.

My system image is mounted with the following:

sudo mount -t ext4 -o loop ~/arm_builds/TK1/system.img /L4T/TK1

Prior to script launch, I run

mount -t binfmt_misc none /proc/sys/fs/binfmt_misc

As per https://www.kernel.org/doc/Documentation/binfmt_misc.txt
because otherwise this error occurs:

../enterTK1.sh: line 6: /proc/sys/fs/binfmt_misc/register: No such file or directory

Now when the script attempts to chroot into my arm system image, I get the following:

chroot: failed to run command ‘bin/bash’: No such file or directory

However /L4T/TK1/bin contains an executable bash. Could this be an issue with qemu or my kernel not reading binfmt_misc/register correctly and redirecting to qemu?