OTA udpate fails with "Cannot write: No space left on device"

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hi,
My devkit Orin does not have enough eMMC space for OTA update from 36.6.3 to 36.6.3.
I successfully updated from 35.5.0 to 36.6.3,
but now I need to re-release 36.6.3 because of some bugs in previous installation,
but it simply does not fit!

My 36.6.3 installation has a custom system.img, which is based on stock rootfs with only few components added, the largest of them is nvidia-jetpack. Orin is kind of useless without Jetpack, so I have to install it.
But it is huge, once installed it takes about half of the rootfs partition:
/dev/mmcblk0p1 56451232 22706304 30844960 43% /

Then I downloaded APP partition to system.img using flash.sh and followed
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/SoftwarePackagesAndTheUpdateMechanism.html#preparing-for-an-image-based-ota-update
: prepared OTA using
sudo -E ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -s jetson-agx-orin-devkit R36-3
It generated ota_payload_package.tar.gz - 11 GB in size.
Then I transferred it to /ota and ran
./nv_ota_start.sh /ota/ota_payload_package.tar.gz
and it failed with
‘tar: Linux_for_Tegra/tools/ota_tools/version_upgrade/ota_validate_params.sh: Cannot write: No space left on device\n’
Indeed:
df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mmcblk0p1 56451232 56434660 0 100% /
du -sh /ota/
11G /ota/
du -sh /ota_work/
22G /ota_work/

So, after installing Jetpack I had 30 GB left, ota_payload_package.tar.gz took 11 GB and /ota_work/ filled the rest.
So, OTA procedure in DeveloperGuide simply cannot work on Orin with Jetpack installed.
I wonder whether it can be changed somehow?
May be I should create a TMPFS in RAM and put /ota directory with ota_payload_package.tar.gz there? Will it work?

I see this post: https://forums.developer.nvidia.com/t/during-ota-update-no-space-left-on-device/ but it appears to be unrelated.

Thank you

4

Hi jhnlmn,

Can you install nvidia-jetpack through commands after you finish the OTA update to fix the bug?

Sorry that I am not clear about this.
Do you mean that you clone the APP partition from your board to generate system.img for OTA update?

Please share the following 2 logs for further check.

  1. full log when you are generating OTA payload from host
  2. the log when you run nv_ota_start.sh on your board
5

orins in the field may not have Internet, I would have to provide my own hosting for all .deb files, which is too much trouble. One atomic system.img update or single OTA update is much simpler and safer to use than multiple .deb files.

Yes. But I think that any other method of generating custom system.img will also fail as soon as system.img exceeds 11 GB.

I attached the logs, but the problem is clear: there is simply not enough space on the device for system.img of 11 GB or larger (once I was able to squeeze by deleting all logs, temporary files, etc, but it is not reliable).
This 11 GB is stored 3 times:
/ota/ota_payload_package.tar.gz
/ota_work/ota_package.tar
/ota_work/internal_device/system.img
total: 33 GB
and it is a bit larger than free space on Orin with Jetpack installed .
l4t_generate_ota_package_2024_07_08.txt (612.7 KB)
ota_update_no_space_on_device_2024_07_09.txt (7.2 KB)

6

Have you considered to add NVMe drive on your device ?

7

Have you considered to add NVMe drive as a standard part to your devkits?
You have to, otherwise your OTA procedure does not work. I am just the first person who tried it with 6.3.0. And note that I did not install any of my own software to rootfs (only Nvidia’s packages). As I understand, other people are creating much larger rootfs with a lot of custom stuff.

But for now I found a work around, as I suggested before, to put /ota on tmpfs/ramfs:

mkdir -p /mnt/tmpfs
sudo chown $USER:$USER /mnt/tmpfs
sudo mount -t tmpfs -o size=16g tmpfs /mnt/tmpfs
mkdir -p /mnt/tmpfs/ota
rm -rf /ota
ln -s /mnt/tmpfs/ota /ota
sudo chown $USER:$USER /ota

It seem to work, but it will be nice if Nvidia test and validate this approach.
Thank you

1 Like
9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.