Can anyone help me ?
I want to use flash.sh to flash into development board with some software, such as vscode, cuda, and the third-party.
How to achieve it? Any guidance would be appreciated!
This is actually a much more useful and complicated question than it seems (and simultaneously, simple if you understand account owner and group, a topic of backup and restore). It’s quite useful to explain this in detail. That detail is only a small bit about how flash generates an image, and much more about how user and group IDs work. This is long, so feel free to ask details about specific content you might be working with.
During a normal flash mostly everything comes straight from the “Linux_for_Tegra/rootfs/” subdirectory. Regarding what does not come from that, it depends on the flash target. Notice if you run the command “ls -l *.conf” from “Linux_for_Tegra/”, that each of those is a flash target. The symbolic links are just more common names pointing at specifications which are a combination of module plus carrier board. A series of files include others, and they are all human-readable. So you could make a custom target, but the purpose for pointing that out is something else.
Any give target has a specific boot content which is copied into the rootfs/ before generating an flash image. Some of that includes:
- The “
rootfs/boot/Image”. - The device tree.
- Possibly firmware.
- The “
rootfs/boot/extlinux/extlinux.conf”. - Perhaps other items, but all related to the initial boot, and not to other parts.
I don’t know what your specific hardware is. A dev kit differs from an eMMC module on a third party carrier board. You will need to specify exactly which hardware you have, but I’ll pretend it is a dev kit, so the target is “jetson-xavier-nx-devkit”. A flash command might be something like this:
sudo ./flash.sh jetson-xavier-nx-devkit mmcblk1p1
If you wanted to log this (and I very highly recommend doing this once so you can see exactly which files are copied into “rootfs/” while creating an image) you could append " 2>&1 | tee log_flash.txt" and you’d know exactly what is used for the flash. The same command with logging:
sudo ./flash.sh jetson-xavier-nx-devkit mmcblk1p1 2>&1 | tee log_flash.txt
If you are just copying files, it is rather simple, although there are a few details to know first. Technically, it is as simple as copying files into rootfs/, and if they are not part of /boot (check your flash log when flashing your specific target board), then any file would normally “just work” and exist where you expect it to be after a copy into rootfs/. Aside from the mentioned files of boot, device tree and firmware, everything is created verbatim in the final creation of the file system, including numeric user and group IDs.
There are some twists in this though, so I will emphasize that the content in “rootfs/” is essentially for a different computer stored on the host PC (think of rootfs/ as a remote backup), and that accounts will very likely differ between the host PC and the Jetson. When manipulating files, the host PC will want to think you are talking about its accounts, and not the Jetson’s accounts. User ID and group ID are actually just numbers, and the name you see is an alias for that number. If the number does not exist on the host PC, then you still need that numeric ID to be valid on the Jetson (and this won’t happen without extra effort, you must specify numeric ID preservation). If the numeric ID has a different name on the host PC, then it is ok that the host PC sees a different name, it is just an alias; realize that the name will change on the Jetson, and what you are concerned with is the numeric ID (group and user). If the correct numeric ID is present, then the alias will automatically succeed. Maybe both PC and Jetson have the same details for the content you are copying, and you might not even need to care.
To illustrate, on the Jetson, before any accounts are added, everything has its name-to-number relation shown with (relative to Linux_for_Tegra/):
cd rootfs/etc
cat passwd
cat group
(are there any different accounts? are there same name accounts with a different numeric ID? more specifically, are the accounts on Jetson and host the same for the exact files or directories you will copy?)
You could consult the man page for passwd and group, but my point is that these are mappings from numeric ID to human name alias. The two can differ between Jetson and host PC. If you have a file with user or group being a system account, e.g., root, then the host PC and Jetson will be the same if they are both Ubuntu (user IDs for root and a few other accounts are always the same across Linux distributions, and even more are always the same from one Ubuntu release to the next). Mostly. Files copied into rootfs/ are rarely an issue if they are not part of that boot content, e.g., if they are not these files:
/boot/Image- Perhaps kernel modules which go with that
Image. - Any
/bootdevice tree. - Sometimes other firmware.
/boot/extlinux/extlinux.conf
It is the numeric user and group ID which will change if not careful. User root (using sudo) has the ability to copy files with a constant numeric ID (and files of different users). Root is able to copy and create files of an ID which does not exist on the local computer, but perhaps exists on the Jetson. A normal user can only copy files belonging to themself. Even root must specify to preserve numeric ID in many cases, and if not, then a translation is performed (you do not want any ID translation).
When I build more than one computer to use with Linux I go through the extra effort to be certain that all of my users have the same numeric user and group ID. That means it is ok for me to simply copy files back and forth and never care. You won’t have that luxury unless you got lucky. But your first user is likely the same numeric ID as the first user of the Jetson even if their names differ. All of my end user accounts on Jetsons and my main computer have the same numeric ID, which was intentional, e.g., my admin user has UID and GID 1000. I have some “adjunct” accounts, e.g., nvidia and ubuntu at times, and these have a different UID and GID, but they were intentionally kept constant across all systems such that I don’t need to worry about this during any kind of copy (I have no need for rsync or sudo on most file operations across systems). Check this command on both host and Jetson, and if the numbers match, then regardless of names, your copies will work as expected (the host PC would show a different name, but the name would go back to what is expected when flashed to the Jetson):
id
All of the above is more or less about backup and restore. Information you find about using rsync to back up or restore files is likely valid for what you are asking about (rsync does work with device special files, sockets, pipes, so on, and is more than just a copy with numeric ID knowledge).
When an Ubuntu package (really a Debian package, it is that mechanism) is installed, you can expect IDs to be correct. If you were to use QEMU to install packages on the rootfs/, then you don’t need to worry about numeric IDs. It is possible to create a user ahead of time on the host PC via:
sudo ./tools/l4t_create_default_user.sh
That command uses QEMU to create an account within the rootfs/ using a bit of “Jedi trickery” (these are the droids you are after! 😀 ).
“In the old days” (I must be ancient), before JetPack, one only had command line flash. The procedure is still valid, JetPack is just a front end to that, but one would do this to install flash content on the host PC:
- Unpack the “driver package” as a regular user. This creates “
Linux_for_Tegra/” with an empty “rootfs/”. - Unpack the “sample root filesystem” with
sudoin “rootfs/”, which creates a 100% Ubuntu rootfs without the needed boot support or drivers. - Run “
sudo ./apply_binaries.sh”, which is what applies the NVIDIA boot and driver content withinrootfs/(this is when the rootfs changes from being purely Ubuntu to instead being “Linux for Tegra”, or L4T).
You can still go to your L4T release (see “head -n 1 /etc/nv_tegra_release”), download just the driver package, and experiment. Go to some unused temp empty directory, then unpack the driver package as a regular user, followed by unpacking the sample rootfs in rootfs/ using sudo. Now comes the magic, where you will want to log it to see how it works:
sudo ./apply_binaries.sh 2>&1 | tee log_apply.txt
Check how QEMU is used in that. Then add a user which would be present during flash to avoid first boot account setup, and log this too:
sudo ./tools/l4t_create_default_user.sh 2>&1 | tee log_user.txt
If you understand the QEMU content, then you can use that to run commands directly inside the rootfs/ as if it is a running arm64 system. Not usually needed if you already have files, but if you need to change packages on the fly, this could be useful to you.
Switching gears to using rsync, it hasthe option “--numeric-ids”, and combined with sudo, this can do most of what you want when there is a difference in accounts between host and Jetson. For more rsync info:
https://forums.developer.nvidia.com/t/how-can-i-quickly-copy-a-system-from-one-tx2-nx-to-another-tx2-nx/276071/2
A less verbose answer would require knowing more about the user and group, and whether they are the same on host PC.
Some simple examples:
- If you copy a file named “
deleteme.txt” to “rootfs/” or somewhere else, then after flash it will exist just as you created it. - If you clone a Jetson rootfs, and loopback mount it on “
rootfs/”, then that clone will be altered with the kernel and boot changes specified in the target, but otherwise you will get a 100% exact match of the original system (e.g., package updates you’d performed will exist, new accounts added will exist, home directories will be copied in, so on). A clone preserves numeric IDs. - If you
rsynca loopback mounted clone (using “--numeric-ids”) to the “rootfs/”, then you get the same thing as mounting loopback there, except that your clone remains unchanged.
See the rsync info for more information on backup and preserving permissions if you will use that mechanism. If you need more information on a specific case, mention where the files are and if they are system accounts or user accounts owning them, so on.
2 Likes
Thank you very much for your thoughtful reply. It took me some time to understand your answer, please forgive my limited ability, communicating with someone not tech-savvy might be uncomfortable for you. I might not have fully understood, so let me briefly describe what I have done following your guidance and some shares from forums:
- Install QEMU:
sudo apt-get install qemu
sudo apt-get install qemu-user-static
-
Extract the Jetson Root File System:
sudo tar -xvpjf Tegra_Linux_Sample-Root-Filesystem_R35.3.1_aarch64.tbz2 -
Copy the QEMU ARM Emulator to the Root File System:
sudo cp /usr/bin/qemu-aarch64-static <rootfs>/usr/bin/
This command copies the QEMU ARM Emulator (qemu-aarch64-static) to the /usr/bin/ directory of the extracted ARM root file system. This allows running ARM architecture programs in the chroot environment. -
Bind Mount System Directories:
cd <rootfs>
sudo mount /sys ./sys -o bind
sudo mount /proc ./proc -o bind
sudo mount /dev ./dev -o bind
These commands bind mount the current system’s /sys, /proc, and /dev directories to the corresponding directories in the ARM root file system. This is necessary for correctly accessing these resources in the chroot environment.
-
Enter the chroot Environment:
sudo LC_ALL=C chroot . /bin/bash
This command uses chroot to switch to the ARM root file system environment, while opening a bash shell. This way, you can run commands in the ARM environment. -
Install Packages in the chroot Environment:
sudo apt-get update
sudo apt-get install <PKG_NAME> // such as cuda
-
Set Default Username and Password (Run before flashing):
sudo ./l4t_create_default_user.sh -u pcai -p pcai -
Flashing:
sudo ./flash.sh jetson-xavier-nx-devkit-emmc mmcblk0p
After completing the flashing process, I am unable to start the system. I’m having trouble pinpointing where the problem lies. Below are the logs recorded during the startup process. Could you please help me with this? Thank you.
Here is the error log:
You need sudo ./apply_binaries.sh between Step 2 and Step 3.
Also, it’s mmcblk0p1, not mmcblk0p.
1 Like
Thanks for pointing out my error！
But The same error occurs after applying the changes.
你的the same error是什麼？你有把整包BSP刪掉重來？
還有log麻煩用serial console抓文字檔，不要用截圖的
抱歉，系统烧录完启动不起来，目前用不了相关的控制台获取log,我只能从显示屏看到相关的log
depmod: ERROR: could not open directory /lib/modules/5.10.104-tegra: No such file or directory
depmod: FATAL: could not search modules: No such file or directory
ERROR: mmcblk0p not found
bash: cannot set terminal process group (-1): Inapproppiate ioctl for device
bash: no job control in this shell
我现在重新生成了image可以正常启动了。十分感谢。
I assume that was just a typographic error, but the destination location is mmcblk0p1 (it was missing the trailing 1). Ahh! I see @DaveYYY beat me to that already 😀
As mentioned by @DaveYYY you also need to apply_binaries.sh before you make any other modification. it is possible that some modifications might work out of order, but it is asking for unexpected issues if done in the wrong order.
A comment on the missing directory, which might indirectly help:
/lib/modules/5.10.104-tegra
A kernel has a specific release, but during compile it has parameter CONFIG_LOCALVERSION also set. When you run the command “uname -r” it is the version with CONFIG_LOCALVERSION appended. So the kernel Image was version 5.10.104, and during its compile it had CONFIG_LOCALVERSION set to “-tegra”. The result is that “uname -r” replies with “5.10.104-tegra”.
What makes this important is that a kernel always wants to find its modules in:
/lib/modules/$(uname -r)/kernel
With that missing it implies the modules won’t be found.
Now I’m going to slightly switch gears and describe part of the flash process which is closely related to this…
Whenever you designate a flash target on command line, such as jetson-xavier-nx-devkit-emmc, and mmcblk0p1, you are specifying a kernel, an extlinux.conf, a device tree, and some of “/boot/extlinux/extlinux.conf”. Flash will then copy in the correct kernel, and put it in the required location of rootfs/. This is why rootfs/ is not an exact 100% match to the changes you make there. Everything except that content is a 100% exact match. If the last time you flashed you used the same parameters, then even the /boot content is an exact match because it copies the same kernel in and the extlinux.conf is the same.
A reason for extlinux.conf to change is if the boot target differs. The default on an eMMC model is the eMMC as rootfs. If you were to specify mmcblk1p1 (an SD card model, which works on SD card models but not SD card models), then extlinux.conf would change, along with device tree. The main kernel would be the same in either case most of the time. If your boot target includes an initrd, or any alternative boot media, then you will find there are multiple “/boot” directories…one on each media. You don’t always use the “/boot” which is the main filesystem for “/”, sometimes the “/boot” of other media is used prior to mounting “/”. Sometimes kernel modules also use a different location, namely if an initrd is run before transfer to the final rootfs.
As an example, an eMMC boot uses the “/boot” and modules on eMMC. An initrd boot to an NVMe uses the “/boot” of eMMC, along with the extlinux.conf of the eMMC, but the modules from the initrd. After mount of the final rootfs, it is possible that the “/boot” is from the NVMe, but it isn’t the one the kernel booted from. Also, after an initrd completes, it performs a pivot root (or equivalent), that makes future module load from the NVMe of this example, and the initrd itself ceases to exist (it was just in RAM).
When you boot to eMMC with no external media and no initrd, you can expect this to have “/lib/modules/$(uname -r)/”. However, if you did not use your flash software to flash the eMMC target at least once, then no kernel (nor modules) would have been copied into “Linux_for_Tegra/rootfs/”. It is the flash target at the moment of flash which copies that content in.
The conclusion is one of these:
- Command line flash did not specify an operation which copies content into “
rootfs/”, e.g., no image was generated (copy is just before image creation). - You used an
initrdor external media, and the content is there, but it is on a different storage device, e.g., on eMMC when looking at NVMe.
On the host PC, if you go to “Linux_for_Tegra/rootfs/lib/modules/”, do you see subdirectory “5.10.104-tegra/”?
Incidentally, mmcblk0p is incorrect, it is mmcblk0p1, and this could result in failing to copy the module content. Remember how I said that the flash target determines some of what gets copied into place? If you really did miss that trailing 1, then your target changed.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.