Jetson Xavier Boot on SSD with Cboot

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hello guys,
I personnaly have a Jetson Xavier, and with some try, I succeeded to flash with the sdk.
(Python 2.7 was needed on the Host).

Currently, on Jetson Xavier , U-boot is not used , which allowed to build kernel and boot on ssd easily on TX version, modifying the extlinux.conf.
C-boot is used instead, and developper NVIDIA fan as me, tries to implement the SSD integration with boot on Xavier version.
This feature was not planned on the product apparently.

This is quite paradoxal for this kind of AI beast product.

Did guys have success to boot on ssd and extend the memory?

I really hope this feature could be solved in the future.

2

Hi guillaume.morinduponchelle,

This feature is still not planed to implement on Xavier yet, would you please elaborate more about the use case and the importance? Then can turn to ask our develop team internally if have chance to implement it.

Thanks

3

I would guess the reason of booting from the SSD would be the minimal speed boost and the larger drive space. I too installed a 1TB SSD that would like to have it bootable, as of right now, itā€™s mounted as the /home folder.

4

Hi kaycc, Hi Ionut,
Thankā€™s you for your reply.

I already mentionned this issue to Serge Palaric, Vincent Palaric, Eddie Seymour and Shannon from Silicon Highway.

As Ionut mentioned, I can see 3 main reasons :

  • First and the main, itā€™s too extend the space in a natively way. I mean extend the ā€œ/homeā€ and the ā€œ/ā€ file system to avoid to be limited on the number of software you can install on the file system. It could be sad if the only limitation of the Xavier, is the storage use ^^.

  • Second, booting on SSD against eMMc offer better performance on reading/writing operations.
    eMMc : 100Mo/s reading, 50Mo/s writing.
    Against
    SSD NVme Pro : 3500Mo/s reading, 2700Mo/s writing.
    You multiply by more than 20 times the perf.

  • Third, on the previous U-boot system, you were able to modify the extlinux.conf if I remember well. Thatā€™s mean you were able to configure it to boot in priority on the SSD and if something goes wrong to boot on the eMMc and reconfigure your SSD. The role of backup is, in my opinion, very important too.

  • Fourth (bonus ^^), itā€™s to use completely the SSD feature as a(n) homogeneous system. This feature also renforce the added value of the product.

Am I the only person who are surprised about this issue ? XD

Ionut, if you have a solution to extend the /home folder, it could be a good starting point.

Thankā€™s you guys.

1 Like
5

If the drive is already partitioned as ext4, you can just mount it, move the home partition to it and modify the fstab.

sudo mkdir /mnt/home
sudo mount /dev/nvme0n1p1 /mnt/home
sudo cp -a /home/* /mnt/home/
sudo rm -rf /home/* (make sure itā€™s backed up at /mnt/home)
sudo umount /dev/nvme0n1p1
sudo nano -w /etc/fstab

In /etc/fstab add:
/dev/nvme0n1p1 /home ext4 defaults 0 1

sudo mount -a

Make sure /home looks good and itā€™s mounted, reboot the system, once booted and you verified the /home is good, delete /mnt/home and that should be it.

6

Hi,
Thankā€™s you for your steps, I had the same issue.
It could be professional and great that Nvidia provide a clear solution.

Thankā€™s guys for your help.

1 Like
7

Yes, I am also very interested in this functionality. It makes development on the Xavier as easy as development on the Nano. On the Nano I boot off of an external SSD drive (2 TB Samsung).

This makes ā€œcreating my development environment for all phases of developmentā€ possible - i.e. before I hone my eventual final production/release code and environment for ā€œreleaseā€ (whatever ā€œreleaseā€ is - it might be a robot that I am constructing, it might be anything else).

For instance:

  1. I built and installed Visual Studio Code so that I have a usable IDE on Xavier/Nano.
  2. I built and installed LLVM and clang v8 so that I can have the latest CUDA support from that compiler as well as the latest and greatest C++ standard/support there.
  3. etc. As we understand serious software development may require a long and intensive and experimental development process. Along the way you will install many tools that need general availability - i.e. ones that it is annoying to attempt to store in the /home directory or some other drummed up non-standard directory.

29.5 GB of usable eMMC card space is just not enough space for the development phase.

On the Nano this is easily circumvented by PIVOTING THE ROOT to an external SSD. I have a root partition with 2TB of space on my Nano. It is quite unfortunate that the Nano is by far the slowest of the Jetsons - I would much rather have this 2TB root on my Xavier which has some real processing power. Now there is still a limitation on the Nano pivoting because the uboot ā€œchoose boot systemā€ prompting is apparently there but isnā€™t visible on the screen. Also the keyboard driver is not activated on the Nano at this point so you cannot enter a manner of booting. Apparently if you use the UART serial connection you can actually interact with the uboot choice prompt - I havenā€™t tried this yet.

As well you have awesomely provided an M2 slot on the Xavier which makes it even more natural and convenient to use the NVMe SSD as the root partition.

It seems you have gone 95% of the way to providing an awesome system with a supported PCIe NVMe SSD M2 slot, but you have left the root partition on the eMMC card and thus pretty much crippled the potential. My $99 Nano is a better development machine in this regard - and I spent a lot more to own the Xavier.

Is there a manner by which I can rebuild the kernel with uboot enabled and thus enable pivoting the root using extlinux?

Thanks and sincerely,
David Bien

8

Hi David Bien,

Well, I think you said everything ^^

Jetson Nano is -far- more maker friendly.

Jetson Xavier is paradoxally for Industries but have a very small amount of storage.

I wrote a lot of mails exchange with the person involved on the Tegra Architecture, Vincent Nguyen. And the exchanges was more a deaf dialogue.

I can share the answer I had from the person involved on the Tegra Architecture after I explained all the advantage from SSD/ eMMC boot.

#BEGINNING#
"
Le 2019-04-16 17:07, Vincent Nguyen a Ć©crit :
I know full well the speed advantage of SSD over emmc.

What I donā€™t quite get is why you absolutely need to have boot
support at bootloader level.

There are essentially three options:

1/ Plug in SSD through NVMe, bootloader, kernel and filesystem on
eMMC, all applications executables and data on NVMe SSD, only downside
is that OS filesystem speed is limited by eMMC speed, but given that
most IOs will be on NVMe SSD, I donā€™t think this has a huge impact
(certainly no other customers have complained about it)

2/ Keep bootloader and kernel on eMMC, modify kernel to load
filesystem from external NVMe: theoretically possible (same approach
as booting from NFS for that matter), although Iā€™ve not looked into
it for ages. Since bootloader and kernel are loaded once, gets the
benefit of advanced IO speed for OS filesystem operation, but I
honestly doubt of any significant real world benefit over 1/

3/ what you are requesting, ie putting kernel and filesystem on SSD,
requires bootloader modification: the only advantage you gain over 2/
here is to maintain coherency between kernel and filesystem, but that
point IMO is moot since you also need to maintain coherency between
bootloader and kernel.

Also, what is the rationale of ā€œit is not suitable to use external
SSD over NVMeā€, if all your applications executables and data are
precisely on NVMe SSD ? Even if itā€™s, I quote, ā€œplugged in like
USBā€, if itā€™s faster why would it be a problem ? Do you have a
specific use case in mind that clearly exhibits a performance
limitation linked to 1/ or 2/ ?

Granted OS filesystem operations (libraries loading, etc) would be
slower when using 1/ above than when using 2/ or 3/, but since you can
also set the swap file to an external device (in this case NVMe SSD),
this would be significantly alleviated after first load. On top of
that, most of the IO will be linked to the application itself
(database would reside on NVMe SSD for ex), not the OS itself.

I will escalate the issue internally though, but I will need some hard
data for engineering to actually invest resources into it, so if you
have a clear use case with data that highlights the performance
impact, I would welcome it (you mentioned you implemented it on TX2, I
would assume you have some numbers if there is such a use case ?).

Regards

Vincent
"
#END#

9

Iā€™d like to add that it is very convenient to have different images on different SD cards. It makes debugging issues easier and also greatly simplifies delivering software to customers.

For example I could have one SD card devoted to debugging a camera issue and a second devoted to working on some user-space application. I can switch between the two without spending a lot of time configuring the environment. Having multiple SD cards is less expensive than having multiple Xaviers!

Having the entire system self-contained makes deployment and upgrades easier too. We have had situations where customers had important demos with very short deadlines. The customer had difficulty updating the system so we ended up overnight our complete working system. Sending an SD card would have been much easier. Instructing a customer to simply pop-in an SD card is a nice, simple solution that isnā€™t error prone.

Thanks,
Greg

1 Like
10

One more reason, to have it bootable: on industrial platform we need scalability, right? We canā€™t waste our time flashing all the xaviers via Jetpack. Thatā€™s just not possible if you have hundreds of them.
Obvious solution is to clone the disk and just insert identical disks into fresh identical xaviers. So worker on assembly line will just grab xavier in one hand, grab cloned ssd in another, sounds of hammer - shazam! we have a product. We have 99% of this implemented, except one tiny bit - bootable ssd.
So we can do prototyping, but we canā€™t proceed to production.

THAT MAKES XAVIER ABSOLUTELY POINTLESS FOR INDUSTRIAL USAGE. Itā€™s just a cool toy :'(

11

Booting from the SSD on a Xavier would be a nice-to-have. However I fully understand the logic of Vincent Nguyen as booting from eMMC and having applications and data on the SSD is feasible and can be fully automated as well. Most IO is happening to applications and data and not the basic OS in almost all usecases anyway.

In my case the SSD is used for Docker images and volumes - integration of the SSD as part of provisioning can be fully automated in just a few lines of code - have a look at https://github.com/helmuthva/jetson/blob/master/workflow/provision/roles/nvme-ssd-integrate/tasks/main.yml and https://github.com/helmuthva/jetson.

Basic knowledge of Ansible is assumed in this particular implementation.

12

I donā€™t think, that alternative boot strategies are ā€˜nice-to-haveā€™.

If Nvidia really want to target industrial applications then a fail-proof way of dealing with different system images is fundamental!
And in the field there is usually is no USB host available to debug or flash new images.

Today, every low budget ethernet switch has a boot system and different system images (recovery, different firmware versions, ā€¦).
My root webserver (somewhere in Germany) has a recovery system, that saved me many times.

So, we donā€™t talk about ā€˜nice-to-havesā€™ but about serviceability of systems outside the lab.

My system of choice would consist of a configurable boot mechanism with a (flashed) recovery system in EMMC (including certificates) and at least two working images on SSD: one for the running system, one for the ā€˜newā€™ system.

Best
Peter

13

Dear Nvidia team,

Is the decision to implement it (boot from SSD) still pending ? Iā€™m driving a project that relies on it and not having access to this functionality could lead to consider non-nvidia alternatives.

14

Just bought and installed Jetpack onto the Xavier, including a really fast NVMe SSD and 10GbE ethernet. At this point Iā€™m completely flabbergasted that you still cannot boot from NVMe SSD at this point in time.

So consider the following as the use cases and importance ( I could NOT agree more with Peter!)! Oh and another use case kuch Tesla eMMC ā€œgateā€ kuch. And now implement it! Just do it: it is really a no brainer in my honest opinion. I do suspect that the performance of Xavier with formal NVMe boot support is a little bit to good.

And yes I have another very strong use case of my own (in concept phase). It involves: Machine/deep learning (Xavier, Nano), data science, predictive analytics, business intelligence, domotica, IOT, mesh wireless protocols, sensors, cameras (including heat and infrared), Kubernetes /KubeEdge, medical devices, complex intensive child healthcare @home (yes really IC @home is quite common in The Netherlands: you simply are not allowed to stay in hospital). Parents do have nothing to help them out. All of them reinvent the wheel themselves, mocked about by the healthcare insurance companies and ridiculed by the numerous doctors/nurses/other healthcare professionals.
My business is to link all data sources to support parents and/or caretakers. Digital medical dossier of a patient in The Netherlands (academic) hospitals is hilariously outdated and dominated by the principle healthcare by obscurity (really just a scan of a doctorā€™s letter/document all of them put together forms a medical dossier). Families are ripped apart and even children/elderly do die unnecessarily do to the prevailing old-fashioned conventional way of thinking that healthcare by obscurity is the way forward. In my use case Jetson Xavier machines are really really important. Although in concept phase, this is going to be very big. Proof of concept sponsored by NVidia?

So it is of fundamental importance and essence of a fail-proof way of dealing with ā€œdataā€ regarding child healthcare at non-formalized-ICU locations. Furthermore, privacy of children/parents/caretakers is a real thing! Therefor most of the data must stay at home. Consider it as the personal medical dossier. Besides that Xavier/Nano are also really ideal form factors for my use case!

@ Peter: fantastic piece of text and in all honestly I couldnā€™t write it better.

15

Are there any news regarding boot process configuration or multi-boot capabilities?

I have the impression, that Nvidia underestimates the importance of this ā€˜featureā€™.
We will now evaluate other platforms that are more flexible (and reliable).

16

Hello,

With some tweaking to the CBoot source, rootfs can be moved to the NVMe SSD and also be used to boot different kernels during boot time. I have not tried with the latest CBoot source but this was true with the one shipped with R32.2.1.

17

Dear Anish,

Do you have any instructions on how to do it? I am very interested in any working solution as Nvidia is obviously not quite getting the point.

18

Hello,

I still have not tried doing this. Sorry about that. I will go through the CBoot_src and see if thereā€™s any CONFIG_ option to turn on SSD support.

However, I do see an option ā€œnvmeā€ in the cbo.dts file which means that the latest cboot may have added support for booting off of the NVME.

19

Hi kayccc!

In our case reason is simple.
Right now we are testing Xavier AGX with the idea of using about 250 AGXs (non dev kit) in production this year. Next year around 500.

Setting up every single device is extremely inconvenient, so cloning full system is the way forward.
But if cloning is out of the question, so is AGX, and weā€™ll have to find the alternative.

20

Please refer to NVIDIA Jetson Linux Driver Package Software Features : Flashing and Booting the Target Device | NVIDIA Docs

For any boot related issue, please open a new topic.

Thanks

1 Like