allspark@tegra-allspark:/sn$ uname -a
Linux tegra-allspark 5.10.216-tegra #1 SMP PREEMPT Sun Mar 23 14:35:10 EDT 2025 aarch64 aarch64 aarch64 GNU/Linux
As the Topic states, after our OTA software update is complete from invoking the ./nv_ota_start.sh as follows:
nv_ota_start.sh /ota/ota_payload_package.tar.gz
So after nv_ota_start.sh is done running, I would like to browse the reulting uprgaded partition before we power cycle the board.
Hi Jerry. Here’s whats happening. After an OTA update, we power cycle the board. Upon bootup into our newly updated partition, we’re seeing ome weird stuff in the file system. The following 3 things are the most common of the strangeness of what am seeing.
Our OTA file that still sits in the /mnt/crypt_root_other partition is now corrupt.
A file that should be there is … is not there.
A file that should not be there … is there.
My most immediate concern that is causing some issues is the 3rd entry above: a file that should not be there is there.
So after the OTA update is complete (from running the ./nv_ota_start.sh script) I would like to browse the newly updated partition to see if it’s there. What I’d like to know is, is the file there immediately after running nv_ota_start or does it appear during the booting into the newly updated partition. This file is a file that is created once and only once at the first bootup of the board. Once created, it’s 8k in size and is never deleted or modified – it’s there for the life of that board, unless we do an OTA update. But doing an OTA update and booting up into the newly updated partition, the file should never be there until we create a new one after an OTA update. But upon booting up into the new partition, and just before creating a new file, we’re detecting that it’s already there, and it’s zero bytes. This is causing a problem. How did it get there? Who put it there? We code reviewed our app that creates this file. It’s we’re confident that it’s not us. So after we return from running the nv_ota_start script, I would like to browse the partition to see if its there before powercycleing the board.
Funny thing is, it doesn’t happen all of the time. In fact it’s a rare occurence that this is happening. I’ve created an automated set of scripts to catch this problem by doing OTA updates in a loop 24x7. We seeing this maybe on the 2nd or 10th OTA update. Maybe not unitl the 30th or 50th OTA update. Its indeterminate.
So short of picking apart the NVIDIA scripts that are doing the OTA update, I figured I post something here before going down that road.
So if you have any info as to how I can view that newly updated partition before power cycling and booting up into that newly updated partition, please share.
we would like to reproduce this locally, may I also know your step in details,
for instance, did you setup environment variable, such as BASE_BSP and TARGET_BSP?
please see-also Image-Based OTA without layout change.
besides, it’s crucial about how you create OTA payload.
you may also check.. $OUT/Linux_for_Tegra/tools/ota_tools/version_upgrade/Image_based_OTA_Examples.txt
Hi Jerry. I did not have even one minute to gather more info for you today. Tomorrow I should though. But the following I did get to grab for you. I will look into how the OTA package is created tomorrow to make sure it aligns properly with what is epected.
Hi Jerry. I do not belivie that you wil lbe able to reproduce this. We are running on a custom Xavier based board that you will need to reproduce this. That said, if you are still willing to reproduce this error I can see if we can send you one of our systems, but befre that I need to do more digging…
I will start picking apart the NVIDIA scripts that do the OTA update. What I’d be looking for is the exact script and line where the OTA tar file is untarred into the new partition. And then just prior to our powercycling the board, I want to see if the file that is causing the issue is present or not. If you know what script this is in and the line in particular, please pass it on. Otherwise I will don my scuba gear and i’m going in for a deep dive …
PS: One other interesting aspect of this issue is that when this problem occurs, it seems as if OPTEE is no longer functional.
Let me explain.
If, after the power cycle this pin file of ours appears in the file system (of size 0 bytes) that should not be there, we delete it and create another pin file. Afte this OPTEE is inoperative. We can’t log in using the pins that we’ve just created.
However on exactly 1 occasion, I copied the pin file from the old partition in /mnt/crypt_root_other and that worked – I was able to read/write into OPTEE. But that was only on that1 occasion. Every time I have tried that again, OPTEE seems to be corrupted in some way and we can’t access OPTEE at after the powercycle after the OTA update.
Somehow it seems, after we bootup into our newly updated partition, and our pin file is present in the newly updated partition (that should not even be there) containg 0 bytes in it prior to our app that creates the pin file is running yet, OPTEE is out to lunch. This must be related somehow and this is another aspect of this problem that I need to solve.
Hi Jerry. I don’t think I can provide you a simple solution as to how to reproduce this. We have a custon dsigned Xavier board, our software, and what I do to reproduce this problem is I have custom TCP/IP client nd serve software to manage an external power switch to poser cycle the board if the OTA update was a success. If it failed that a FULL STOP and I need to get in to look around to see why it failed.
That said, I may have found a solution and I’ve gotten to 180 successful OTA software updates throuh EOB Monday. Weve never gotten this far since I started looking into this issue 5-6 weekes ago.