In the SDK manager instructions:
https://docs.nvidia.com/sdk-manager/install-with-sdkm-jetson/index.html#installation
it says:
Jetson Nano Developer Kit users need to insert a microSD card into the slot on the underside of the Jetson Nano module, or else the flashing process will fail.
Does this mean it will wipe the SD card or will it leave it untouched?
Thanks.
Yes, it will be reflashed with the version you select to install.
Okay, so it will wipe all user files?
Yes.
Keep in mind that you can save a copy of the SD card, and although the operating system content wouldn’t be something to reuse, any customization you’ve made could probably be put back in. You’d have to describe exactly what content it is though if you wish to do this (it makes advice much better to know what the advice is for before answering).
That is my plan. It is very annoying having to install Ubuntu 18.04, install Nvidia SDK Manager, short the recovery pin on my Jetson Nano just so I can get an up to date SD card set up.
Ultimately, I am doing this because I need 120fps Raspberry Pi camera support in gstreamer and my previous attempts failed because my system seemed “unclean”. If I can’t get 120fps working on a new SD card, at least anyone helping me will be starting from common ground and so will be able to help better.
It isn’t unusual for a newer release of some software being better optimized and faster, and probably the best way to know if one release is faster than another is to try it out. However, reinstalling alone with a new release won’t work miracles. If one used a filesystem which is subject to fragmentation (and ext4 is not subject to this until it reaches rather close to 100% full, plus solid state memory itself finds this less problematic), and you have the same work load, the fragmented system will probably get worse over time and contribute a lot to performance loss if disk performance is part of the metric. Running a lot of unused programs will also be a problem. It seems unlikely though that a Jetson would have degraded performance just from running over a lot of time if nothing has really changed and is using an ext4 filesystem. Optimizing for camera speed is something people post a lot about on the forums, but I think the people who succeed are looking at fine details and getting replies from people who work with very specific circumstances rather than just reinstalling. Be ready to give exact details of the entire pipeline of your video hardware and software in addition to using the latest software.
Very true. I don’t do much disk I/O but if I did I would use eMMC instead of SD card.
At the risk of going off topic (I’m saving my 120fps issue for a forthcoming post if I can’t get it going in the new setup), this is the issue I had previously that I could not resolve:
My system was updated using “apt upgrade” over the course of 2 years. I could see that the kernel bootargs had been duplicated at some point during the time (but not consciously by myself). The Nvidia upgrade process did not appear to be completely flawless. Starting from afresh, I am hoping to apply the helpful suggestions I received to get gstreamer behaving as it used when I bought the dev board.
Until recently there wasn’t any ability to use the “sudo do-release-upgrade” mechanism (Jetsons and many embedded systems don’t have a BIOS, and each boot setup is custom…a bit like flashing the BIOS and bootloader every time you flash; such an upgrade gets complicated). I don’t know which release this actually started with, but it was somewhere in the L4T R32.x series. I can see though the possibility that such an upgrade would in some way work less efficiently than directly installing a new release. Still, it is hard to say without actually performing a fresh install (it is worth trying).
You can keep a clone though in case anything goes wrong. Details differ between SD card and eMMC models, but for the SD card model it might be as simple as using a new SD card and not touching the old one once the dev kit itself has had QSPI flashed (the QSPI has boot content on it, and an SD card might be modified by booting to a different QSPI release…thus, once QSPI has been flashed, if you might want to revert, then make sure the old SD card is not used without flashing the QSPI back to the original release from when it most recently functioned).
The combination of “sudo apt update” and “sudo apt-get upgrade” will itself not be a major issue since the release stays the same and only packages are updated. “Theory” says that if you were to fresh install and run “sudo apt update” and “sudo apt-get upgrade” once, then you’d have the same result as periodically running those commands over time until reaching today’s package releases. In theory, but glitches happen, so it might help to fresh install even in that case.
Boot args is an interesting (and extensive) topic. Once in the Linux kernel though most arguments, when listed twice, won’t matter; only the most recent argument will be used. Some, like consoles, can actually have and use more than one instance (e.g., serial console plus regular console are possible). Why this happens in a Jetson gets more complicated.
Note that in any computer the drivers (and boot stages have drivers too) must be able to find the device and use the correct driver. That means that either the device has to be able to “self report” (be “plug-n-play”), or else be blindly told to use a given driver at a given physical address (along with arguments to the driver). This latter method is exactly what the device tree is for since it allows avoiding putting every variation of a driver and its physical address into the kernel source code.
Take a look at “/boot/extlinux/extlinux.conf”. You’ll see in the APPEND key/value pair (the arguments passed to the Linux kernel), that part of it is usually “${cbootargs}”. This is a node within the device tree. If an argument to the kernel is listed twice in the device tree’s “chosen->bootargs” node, then most of the time that argument will appear twice in the Linux kernel command line as well. This is inherited as an environment variable. Then there is the rest of the APPEND key/value pair in extlinux.conf, and this too has the ability to duplicate something. Only in rare cases would previous values matter, and almost always the most recent value is used. If a value is present which the kernel and drivers are not coded for, then it is ignored.
The reason this exists in the device tree is that the initial hardware setup (including clocks and power rails which might have optional uses, but need to know their specific setup for the given carrier board) can’t query the devices to find out what they are or where they are. In a PC most of this would be part of the BIOS, but there is no BIOS. In a PC the BIOS itself has the equivalent of a custom device tree and small CPU. We get back to the idea that without the device tree there would need to be a BIOS. There would be extra hardware, and that content would still exist, but it’d be updated by BIOS flash instead of boot stages. The device tree “chosen->bootargs” is used for this, possibly edited (the minor possibility of the Linux kernel seeing something different is because of this), and then passed on as the “${cbootargs}” environment variable. I doubt this would have any effect on boot or performance, but it is worth considering that Linux inherits the boot state, and if the boot state differs between a clean install and several updates, such that the rootfs partition is still using 100% exact matches, that it is still possible the inherited boot state will change performance. Not likely much of a change, and it is a slim possibility, but it is possible.
My theory is that there is a problem with my QSPI flash. I realise that duplicated bootargs are not a problem, but they are a sign of potential problems or at least, an update that didn’t go perfectly.
I’ve just updated the Nano and am backing up the new SD card. The Nano boots and the USB networking works and I can login. It does give me some concern that the old SD card had roughly 10 partitions but the new SD card has only 1.
The issue at hand is not one of “performance” per-se but one of having the correct device tree that permits 120fps as a capture framerate. The problem I had before was that the kernel, as booted, was not using the device tree I built, but an internal one (or maybe one that existed on one of the partitions?).
Duplicated kernel command line arguments are rarely a sign of a problem. I don’t see that as a problem. It is mostly pointless to have duplicated arguments, but it has so little meaning that it is rare for various source to check if the argument already exists before trying to add it. What do you see from “cat /proc/cmdline”? Then, what do you see from “cat /proc/device-tree/chosen/bootargs”? And finally, what is the content of your “/boot/extlinux/extlinux.conf” “APPEND” line?
Note that where the replies get the information varies. The “/proc/cmdline” content is purely what the kernel actually sees. However, the extlinux.conf might exist in more than one place (depending on setup; yours is likely just the SD card), and the device tree can be:
As long as the QSPI flash uses the same release as your SD card there is no issue with wiping user data. Make a backup (a clone is highly recommended and easy to create), or just don’t use that SD card with a different flash release. Since you are worried about not being as high of a quality from multiple updates, versus one install and one update, it is best to not use that SD card content again (though you could easily, once done, transfer the “/home” content of one SD card to another, and this won’t hurt the operating system).
I have now solved my problem. The fix to achieve 120fps (and more specifically, to make sure the kernel I just built used the DTB that went with it) was to run the SDK Manager and do a full system update using a blank SD card. I am convinced that this wrote to non-volatile memory somewhere on the Nano because previously, I could not manage to get the kernel to load my device tree. It always loaded a mysterious device tree that I could never track down. :-D
To answer your questions though:
nano@jetson-nano:~$ cat /proc/device-tree/chosen/bootargs
tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,4 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 gpt earlycon=uart8250,mmio32,0x70006000 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 nv-auto-config quiet tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,2 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 pci=noaer
nano@jetson-nano:~$ cat /proc/device-tree/chosen/bootargs
tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,4 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 gpt earlycon=uart8250,mmio32,0x70006000 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 nv-auto-config quiet tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,2 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 pci=noaer
APPEND ${cbootargs} quiet tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,2 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 pci=noaer quiet
(You have to use the scrollbar to view the full text of the first 2 text boxes.)
Although it is solved, I am still curious: Have you modified the “APPEND” part of the extlinux.conf? It does seem quite long, and by default one will usually see "quiet`", but I see this twice. I’m not concerned about it being listed twice, but how it got there says something unusual is going on.
You are right that the boot arguments (from device tree as well as APPEND) are quite long, and this is not really a concern, but once again, for the device tree side, it makes me curious how it grew so long.
If that length of both device tree “chosen->bootargs” and the long length of “APPEND” is a result of package updates, then I’d say you found a non-fatal bug (but still a bug) in the package update path.
No, that is the weird thing. The APPEND line I quoted has never ever been touched by me.
Were the only package updates you performed from “sudo apt update” and “sudo apt-get upgrade”? Those don’t change the release, but could conceivably edit that content. If any kind of update was used other than this (an update which might change the release version), then knowing what that update was would be required for someone wanting to reproduce this.
Also, if it is still available (perhaps you still have the old SD card), you might want to post the “/etc/nv_tegra_release” file on the system with this very long kernel command line.
I definitely used update and upgrade and maybe dist-upgrade. I suspect I also updated my Jetpack manually by following the release instructions. It seems my last update was in July last year:
# R32 (release), REVISION: 6.1, GCID: 27863751, BOARD: t210ref, EABI: aarch64, DATE: Mon Jul 26 19:20:30 UTC 2021
If the dist-upgrade kept the Jetson at the same R32.6.1 release, and if the kernel arguments grew by this, then that is a definite bug.
If the dist-upgrade used a non-NVIDIA method to go from something like Ubuntu 18.04 to 20.04, then this is not supported (but I’d be surprised if such an upgrade would even boot).
Do you happen to remember if this was Ubuntu 18.04 the entire time? Or if the dist-upgrade instructions were suggested by NVIDIA documents, versus from some Ubuntu documentation which was not aware of the NVIDIA hardware?
I only ever followed official recommended Nvidia procedure to update my Jetpack as far as I remember.
I was looking for the official release notes for 32.6.1 but cannot see anything about the way I updated. IIRC, I followed instructions entitled “how to update from a minor release” or something like that.
Edit: always Ubuntu 18.04
Do you remember which release it was prior to R32.6.1? Or if it was a different release prior to the dist-upgrade? This would probably be the only way the device tree chosen->bootargs and extlinux.conf’s APPEND became so long with duplicated values.
No, unfortunately. I just know I never used the Nvidia SDK manager (until I had to recently and the APPEND argument did not change). I had a new Jetson Nano A02 dev board in January 2020 and upgraded it from the command line ever since.
It’s all in the past now anyway.