Creating RAID 10 with 4 USB devices

Hi all,

Does anybody know how to create a RAID10 with 4 USB devices on Jetson Nano?

The following is my way with the error:

# sudo apt-get install mdadm -y
# sudo mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd
mdadm: layout defaults to n2
mdadm: layout defaults to n2
mdadm: chunk size defaults to 512K
mdadm: partition table exists on /dev/sda
mdadm: partition table exists on /dev/sda but will be lost or
meaningless after creating array
mdadm: partition table exists on /dev/sdb
mdadm: partition table exists on /dev/sdb but will be lost or
meaningless after creating array
mdadm: partition table exists on /dev/sdc
mdadm: partition table exists on /dev/sdc but will be lost or
meaningless after creating array
mdadm: partition table exists on /dev/sdd
mdadm: partition table exists on /dev/sdd but will be lost or
meaningless after creating array
mdadm: size set to 244448256K
mdadm: automatically enabling write-intent bitmap on large array
Continue creating array? y
modprobe: FATAL: Module md_mod not found in directory /lib/modules/4.9.140-tegra
mdadm: Fail create md0 when using /sys/module/md_mod/parameters/new_array
mdadm: unexpected failure opening /dev/md0

I don’t have a lot of ubuntu-knowledge , I need your help!
Many thanks for your support!

Regards,
Marco

Apparently, the kernel isn’t compiled with the “md” driver built in, and the module to support the “md” device is also not included by default.
This seems kind-of annoying – I think any linux system should have MD and LVM built into the kernel, but apparently NVIDIA thought different.

You will need to follow the instructions to re-build the kernel, OR the instructions to build a new module for the existing kernel, to build the md_mod module to load into your kernel. Unfortunately, because file system mounts happen early, that module generally needs to be available in the early boot stage, not just user-loaded once the system is up. Which is why I recommend building the kernel with the module static-linked ("configure ‘y’ instead of ‘m’)

First off all, many thanks for your helpful feedback!

I think, I understand you, but honestly I’ve no idea how to recompile a linux kernel which contains MD and LVM.
What exactly do you mean with building the kernel with the module static-linked ("configure ‘y’ instead of ‘m’)
and where can I found the instructions?

Regards,
Marco

When you build something into the Linux Kernel, it can generally be built in to the kernel, or included as a module. If you need to access somethign at early boot time (like to boot off something), it’ll need to be a “y” rather than an “m”.

Basic instructions from nvidia are here but forewarning: this process is not easy.

You should be able to run
“make menuconfig” (or xconfig, if you prefer) instead of modifying the config file by hand after you run
“make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig”
(between steps 3 and 4)

Then you navigate to these bits and change the options you need for raid to “y”

Once you’ve built and booted the system, you’ll then need to “sudo apt install mdadm”, and then set up raid as usual.

Edit: for those new to the thread, there are some scripts now to do all of this automatically:
https://github.com/mdegans/tegrity

I don’t understand it, what does it mean “it’ll need to be a “y” rather than an “m””?
For what stand the “y” and “m”?

Please see the keystrokes section of this article:

FYI, answering “y” puts the driver or content in the actual kernel Image. Ansering “m” puts the content in a module, which is a separate file. Often modules are a better choice, but items needing access during boot, and certain invasive options, typically need “y”. In cases where you can’t integrate a feature required during boot (a.k.a., features you can’t use “y” with), then you have to resort to a lot of trouble creating an initial ramdisk (initrd). Just using “y” is a lot easier, but you might need to rebuild any modules (when you change the base Image, sometimes modules which plug into the Image no longer work, or have quirks).

The Image is part of what determines your “uname -r” output (the suffix is usually from the config item CONFIG_LOCALVERSION, and on a Jetson, usually CONFIG_LOCALVERSION="-tegra"). Modules are always searched for at:

/lib/modules/$(uname -r)/

If add a module, then things are usually a very straightforward file copy. If you change Image, but keep “uname -r” the same, then possibly all modules will work as expected from the old “/lib/modules/$(uname -r)/”, but not always. The Image file is also a file copy to “/boot/Image”, but be sure to save the original Image prior to putting in a new Image.

FYI, I suggest copying “/proc/config.gz” somewhere as a reference to the original running system’s config prior to working on a kernel. A reference copy can be very useful.

Hi all,

I get an error after “make menuconfig”.
Following the actionplan with the error:

  1. Download kernel sources on https://developer.nvidia.com/embedded/downloads

  2. Extract:

     2.1 # tar -xjf <board>-public_sources.tbz2
     2.2 # cd public_sources/
         # tar -xjf kernel_src.tbz2
    
  3. Install the utilities:

sudo apt install build-essential bc

  1. Set the shell variable:

mkdir -p /tegra_kernel

TEGRA_KERNEL_OUT=/tegra_kernel

echo $TEGRA_KERNEL_OUT

  1. Execute the following commands to create the .config file:

cd /home/marco/Downloads/public_sources/kernel/kernel-4.9

sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig

  1. Invoke menuconfig:
    user01@nvidiajetsonnano:~/Downloads/public_sources/kernel/kernel-4.9$ sudo make menuconfig
    HOSTCC scripts/kconfig/mconf.o
    :0:12: fatal error: curses.h: No such file or directory
    compilation terminated.
    scripts/Makefile.host:118: recipe for target ‘scripts/kconfig/mconf.o’ failed
    make[1]: *** [scripts/kconfig/mconf.o] Error 1
    Makefile:565: recipe for target ‘menuconfig’ failed
    make: *** [menuconfig] Error 2
    user01@nvidiajetsonnano:~/Downloads/public_sources/kernel/kernel-4.9$

Does anybody know why I get this error?

Many thanks for your feedback.

Regards,
Marco

Seems you’re missing libncurses. You may check which version is available with:

sudo apt-get update
sudo apt search libncurses

#depending on what is available for your distro, install with something like:
sudo apt-get install libncurses5 libncurses5-dev

Just a simple question: why?

If your goal is: increase system reliability
A simple RAID 1 is enought.

If your goal is: increase storage
A NAS mouted over SSH has more sense, if you think that 1Gbit/s is not enought you can plug more ETH network cards over USB3 and bonding them (obviously, also NAS needs more than 1 ETH port).

PS: this is my personal opinion.

Hi all,

yes, I’ve missed libncurses.
So after the installation of it, I was able to invoke menuconfig.
But I get an other error after step 8: If I’m going to reboot the system, It hangs and I don’t get any output. Do you know why?
Following my actionplan:

  1. Download kernel sources on https://developer.nvidia.com/embedded/downloads

  2. Extract:

     2.1 # tar -xjf <board>-public_sources.tbz2
     2.2 # cd public_sources/
         # tar -xjf kernel_src.tbz2
    
  3. Install the utilities:

sudo apt install build-essential bc

  1. Set the shell variable:

mkdir -p /tegra_kernel

TEGRA_KERNEL_OUT=/tegra_kernel

echo $TEGRA_KERNEL_OUT

  1. Execute the following commands to create the .config file:

cd /home/marco/Downloads/public_sources/kernel/kernel-4.9

sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig

  1. Invoke menuconfig:

sudo apt-get update

sudo apt search libncurses

–> Depending on what is available for your distro, install with something like:

sudo apt-get install libncurses5 libncurses5-dev

sudo make menuconfig

–> add multi-device-support (RAID and LVM) with “Y” --> https://www.linuxtopia.org/online_books/linux_kernel/kernel_configuration/ch09s05.html

  1. Execute the following commands to build the kernel including all DTBs and modules:

sudo make mrproper

sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j4

  1. Replace <release_packagep>/Linux_for_Tegra/kernel/Image with a copy of $TEGRA_KERNEL_OUT/arch/arm64/boot/Image:

sudo cp /tegra_kernel/arch/arm64/boot/Image /boot

reboot

Many thanks for your feedback.

Regards,
Marco

I don’t have time to respond at length as I just got back from vacation and have a lot to catch up on, but it looks as if you may be missing some steps from Nvidia’s instructions to copy the appropriate files.

If you do steps 4-8 on Nvidia’s instructions after the menuconfig and then run the create nano SD card script, you should end up with a bootable system. If it’s your dev system, rather than the nano, that isn’t booting, that shouldn’t be related. Fwiw I don’t see reboot in the steps. You shouldn’t have to reboot your dev machine at all during the process.

@mdegans: Many thanks for your feedback. Yes you’re right, I thought I was finish at step 5.
Now, I’m stucking with step 7 --> I don’t know what to write for the INSTALL_MOD_PATH.
Do you know it?

In addition, which create-nano-SD-card-script do you mean?

So it looks like that command should be:

sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=<whatreverthepathisto>/Linux_for_Tegra/rootfs/

You don’t actually replace “INSTALL_MOD_PATH” with anything. Make sure if you go back after the fact and try to continue that you set TEGRA_KERNEL_OUT again as in step 1.

The create-nano-sd-card script can be found in the Linux_for_Tegra folder wherever you installed sdkm. It’s a script that runs flash.sh with the proper parameters to create a nano sd card. When you get to that point, let me know, or run the script with --help.

Ah okay, so you mean I have to install first sdmk to get this folder “Linux_for_Tegra”?
Then I need to execute the step 5 and 6 from “Building the NVIDIA Kernel” again, because I did it in the wrong way.

How can I install sdkm?

Sorry if these are stupid questions…

Many thanks for your feedback.

Best regards,
Marco

SDKM is short for SDK Manager. Sorry. Should have been more specific.

Until point 4 (inlcuding the menuconfig), I understand it and it works.
But Point 5, 6 and 7 I dont know “<release_packagep>/Linux_for_Tegra/kernel/Image”, what is it?
Point 8 is optionally.

@mdegans: You wrote “You don’t actually replace “INSTALL_MOD_PATH” with anything”, really?

I need help to get raid support on my jetson nano.

SDK Manager is capable of working with more than one Jetson system. That “release_…” is a version number for the module/carrier board. If you’ve used SDK Manager for flashing only a Nano, then there will be only one subdirectory to “~/nvidia/nvidia_sdk/JetPack…Linux…/Linux_for_Tegra/”.

Remember to examine CONFIG_LOCALVERSION (see post #7). If you replace an Image, then you must also replace every single module…unless you set up CONFIG_LOCALVERSION to match the previous Image.

So, I need to use SDK Manager to compile the linux kernel?
I didn’t used SDK Manager to get the linux on my sd-card, I used the following command on another system:

/usr/bin/unzip -p /Users/marco/Desktop/jetson-nano-sd-r32.1.1-2019-05-31.zip | sudo /bin/dd of=/dev/rdisk2 bs=1m

That command is is to flash a binary system image to the SD card. What you need to do is to compile (make binary) a new kernel from source, add it to the appropriate folders inside wherever you installed SDK managed, run the SD card creation script to make your own binary image, and then flash that to the SD card like you did before with dd. In your case, yes, you must install SDK Manager, pick the options for your device, and obtain the kernel sources as is outlined in Nvidia’s instructions.

If you’re not familiar with compiling the Linux kernel, it may be difficult. Nvidia’s instructions seem to be intended towards people who have already done it before.

When I have time I plan to write a script to set things up, prompt the user with menuconfig, and spit out a flashable image. I will try to get around to that by December, but no promises. I need to re-run through the process from beginning to end on a new user, take notes, write it, and fix any bugs first. In the mean time, please keep trying. This isn’t an easy task.

Yes. That command should be “… INSTALL_MOD_PATH=some/path/here …” Like in the code block in post #14

It should be the path to rootfs/lib/modules where you want your modules installed. By defining INSTALL_MOD_PATH, you’re telling the kernel’s scripts where you want your modules installed. The modules will need to match the kernel (“Image” file) version or the system will probably not boot.

The “rootfs” folder, if you’ve found it, becomes “/” on the nano when the SD card creation script makes the image. Any modifications or files added there will be changed on the final image as well. For example, if you wanted to add something in /etc on the nano it would go in rootfs/etc on your workstation.