Flashing from RHEL 9 via CLI

[smcentee]

I need to flash the Xavier NX from a RHEL 9 host (board requirement). Using an Ubuntu VM, I can flash the devkit with the Jetpack 5,1 via CLI (sudo ./flash.sh jetson-xavier-nx-devkit-emmc mmcblk0p1) without issue. After flashing, the devkit boots and goes through normal configuration screens, reboots and presents the Jetson Linux desktop as expected.

If I tar up the whole Linux_for_Tegra directory and move it to a RHEL 9 native machine, untar and can flash successfully. However, the devkit boots, but hangs right before the ‘Applying configuration screen’. It asks me the normal language selection, region, login/password, network configuration, etc. as expected, just like the Ubuntu flash install currently does. Then it hangs. If I reboot the devkit, it brings me back to the start of the configuration screens. Endless loop.

In order for it to run on RHEL, I had to install the LZ4 & LZ4-devel packages for the flash.sh to run. Are there any other packages that I’m missing in RHEL that Ubuntu installs?

Further note, I can take that /Linux_for_Tegra tar file and untar it on another Ubuntu machine (without SDK Manager installed) and it flashes the devkit correctly…so I’m missing something on the RedHat install.

Is the flash software not picking something up while flashing in RHEL that it does on Ubuntu?

Anyone have any ideas? Is there an installation log I can compare between the two? Thanks in advance.

[KevinFFF]

Hi smcentee,

Sorry, we haven’t verified the flash operation on RHEL.
Currently, please just use Ubuntu as your host (actually VM or docker is also not recommended).

@smcentee

However, the devkit boots, but hangs right before the ‘Applying configuration screen’.

For this issue, have you tried to skip oem-config through pre-config them before flash?
You could refer to the following instruction to do this.
Flashing Support — Skipping oem-config

[smcentee]

Thanks for the oem-config skip, that works great in creating a default user (and skipping all the setup screens).

As for using Ubuntu, I cant use that for production, it has to be RHEL in our design. So currently, I can flash from a Ubuntu host PC (via VM with no issues) and boots correctly. If I tar that entire Linux_for_Tegra directory and move it to RHEL and flash, the devkit hangs on ‘Failed to start OpenBSD Secure Service’. Previously it would hang during the setup configuration screens at the desktop level. Is there a way to disable this OpenBSD
service before flashing?

See pic for the error during boot:

[linuxdev]

If you are on a Linux system with sudo access, then normally you could do this to disable prior to next boot:
sudo systemctl disable ssh.service
(you’d lose ssh login)

An interesting query would be:
journalctl -u ssh.service

I don’t know though if you can get to a terminal without messages getting in the way, e.g., ALT-F3.

[smcentee]

When is this done (sudo system disable ssh.service)? Prior to flashing the devkit or while on the devkit while booting? If while booting, how do I get to a prompt to enter this command? It just hangs on the Failed error as soon above and never gets to a login.
Yes, I have sudo access since I set up the accounts on both(host and target).

Thanks.

[linuxdev]

This is performed on a running system (after boot is complete and you’ve logged in as an admin user). If the service is disabled in the rootfs image before flash, then this would work as well, but the default is enabled. It is possible to use an emulator (QEMU) to update this on the sample rootfs image (I have not tried, but the “apply_binaries.sh” script is a good template for doing this since it adds NVIDIA packages to the rootfs via QEMU on the host PC), or to clone a system which is already updated and use that instead of the sample rootfs. A typical place to perform this in case of issues logging in is from the serial console (serial console is unusually reliable since it has so few dependencies). Can you log in via serial console?

[smcentee]

I can not login via the serial console unfortunately. While booting up it gets stuck in at the error (as the screenshot above shows) and never gets to the login. How do I go about disabling this service before the flash? Thanks for the response.

[linuxdev]

What was your exact command line for flash? Is the Jetson a dev kit, or does it use a third party carrier board? Is there any modification to installed software (copying “Linux_for_Tegra/” to somewhere else won’t harm anything).

[smcentee]

I am using the Xavier NX devkit with an eMMC.
My exact flash command was: sudo ./flash.sh jetson-xavier-nx-devkit-emmc mmcblk0p1

I am tarring up the entire /ndivia/ndivia_sdk/JetPack_5.1_Linux_Jetson_Xavier_NX_targets/Linux_for_Tegra directory (the default location from the SDK) install).

Nothing was modified.

[linuxdev]

Are you packaging it such that all file ownership in the “rootfs/” directory is numerically preserved? I’ll explain why not all “copy” of content is valid. I ask because permissions being altered can get in the way of boot.

Every Linux system assigns a numeric ID for each user (see “/etc/passwd”, this is the UID), and a numeric ID for the group (see “/etc/group”, this is the GID). Whenever a command is run, or a file is modified, the name of the user or group does not actually matter. Names are always translated to/from the numeric ID. Numeric IDs are what go into the actual filesystem.

If UID 1000 on a Jetson has the login name of “ubuntu”, but on your RedHat system UID 1000 is user “smcentee”, then viewing that file which shows up as owned by “ubuntu” on the Jetson will show up instead as owned by “smcentee” instead when looking at it on the RedHat system. However, what happens when a numeric ID has a name on the Ubuntu system, but not on the RedHat system? It is a dangling ID. Some form of translation must occur. Or what happens if the backup and move and install steps actually use steps which alter the numeric ID to “fit” an actual user on the RedHat system? Note that there are a lot of IDs in the passwd and group files.

There are many ways to back up or pack a filesystem which preserves numeric ID. It is quite possible that your backup is 100% valid. However, it is also possible that upon mounting or copying into the RedHat system that RedHat itself changes something. You would no longer have a pristine copy.

The user “root” is always UID 0 and GID 0. Everything else is just luck or convention. In some cases certain permission bits require special permission, and either mounting or copying those permissions to RedHat could result in altering those bits unless told to use them verbatim.

In the case of “Linux_for_Tegra/”, it is the “rootfs/” subdirectory which must have all names and IDs and bits preserved. That implies using “numeric IDs” in copy operations (something rsync is good with, but cp is hit and miss on). Many methods will preserve subdirectory IDs, but some will not. I’m wondering if they are now different in the “rootfs/” directory.

Note that you can look at file permissions with “ls -n” and get numeric IDs instead of names. One particular file which often gets changed if the incorrect method of copy is used is the “sudo” file. If you look at “ls -n /usr/bin/sudo”, what do you see? If you again go to your RedHat system, and cd to the “Linux_for_Tegra/rootfs/usr/bin/” location, and then use the command “ls -n ./sudo”, is it a match, or is it different? If they are the same, then probably all is ok.

EDIT: Almost forgot…could you include a log of the flash from the RedHat system? Example to log:

sudo ./flash.sh jetson-xavier-nx-devkit-emmc mmcblk0p1 2>&1 | tee log_flash.txt

[smcentee]

I will try the rsync method instead of the just the cp.

Just to be clear, here are my steps:

• tar the /Linux_for_Tegra directory up on the Ubuntu host PC (sudo tar -zcvf filename.tar.gz)
• rsync the tar file to an externally mounted HD (rsync -av /source-dir /mounted-dir
• move the HD to the RHEL system
• untar (sudo tar -xvf filename.tar.gz)
• flash the dev board (sudo ./flash.sh jetson-xavier-nx-devkit-emmc mmcblk0p1)

Thanks for the flash log command. It is attached for your review to see if anything out of place (this was prior to using the rsync command, just a cp).

Thanks again!
log_flash_RHEL.txt (442.2 KB)

[linuxdev]

For tar, consider using --numeric-owner. For rsync, you’d use --numeric-ids. Past experience tells me that rsync can normally be trusted, but you have to be careful with tar and cp. This still might not be a problem in your case, but if you can’t verify the original media is an exact match to the copied media, it becomes difficult to debug other issues. Make sure your rsync includes --numeric-ids any time the rsync might touch another system (e.g., a second system as source or destination). Also, make sure any numeric UID or GID present is listed in “/etc/passwd” and “/etc/group” (this suggestion because we might be using an Ubuntu system mixed with RedHat).

Something to think about whenever you flash on command line (true for GUI as well, but details differ):

• During flash the kernel, modules, device tree, and extlinux.conf will probably be edited or changed based on the named target board (e.g., jetson-xavier-nx-devkit-emmc).
• Each target board has a related “.conf” file (e.g., jetson-xavier-nx-devkit-emmc.conf).
• The .conf files are human readable, and include others based on the module specification and the carrier board specification.
• You can copy a set of target .conf files and edit them with customizations. You might be interested in changing some aspect, e.g., extlinux.conf or device tree.
• The log you created is basically a map of what the .conf file has in it. For example:
  [ 479.3580 ] Writing partition kernel-dtb with kernel_tegra194-p3668-0001-p3509-0000.dtb [ 323861 bytes ]
  (which says which module and carrier board specification is used, and which partition it went to)

Is any of the content which was specified in the boot log (or the .conf files of that specification) being replaced or edited in a way which is incompatible with the RH image?

Note that if you use the “-r” to “reuse” a system.img, then you are guaranteed no edits occur there. However, you don’t know if partitions were perhaps edited. Compare what you expect to be flashed with the flash log (ignore anything except partitions if you use the “-r” option).

Also consider creating a copy of the .conf file with a custom name, including the recursively included “other” .conf files.

[smcentee]

Thanks @linuxdev for the response. That worked great.

Using the method of capturing the logs you suggested, I was able to compare the two systems and determine that some of xfers were not the same sizes.

So, using the combination of tar --numeric-owner & rsync --numeric-ids flags, I was able to tar up my Ubuntu host PC directory and xfer it to RHEL 9 and flash, without issues this time.

Many thanks again for all the help.

[linuxdev]

One more note: For the size of the rootfs partition, you can specify with the “-S <size>” option. It isn’t directly a byte size, but basically it is a multiple of either “1024x1024” (MiB) or “1024x1024x1024” (GiB). Rootfs partition size (the raw system.img.raw, not the sparse image) would have to exactly match that. For example, if the rootfs raw image is 28GiB, then that means the actual partition size is:
1024x1024x1024x28 = 30064771072 bytes
…and the flash would have this in it:
sudo ./flash.sh -S 28GiB -r jetson...

You can find partition size definitions inside of other configuration files which are picked based on the target board. Should you need a custom version of the boot content partitions, then you’d also have to make sure the partition is sufficiently large to hold both the data and signature.