PCIE errors with Sata controller

Robotics & Edge Computing Jetson Systems Jetson TX2
1

Hi there!

I’m using a Jetson TX2+dev board in combination with: https://www.amazon.nl/gp/product/B07PJFZRRW/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1 a Sata controller, in order to hook up to 4 sata HDD’s to the jetson.

I have hooked up one SSD via the sata connection on the dev board, and I’ve hooked up two 4tb HDD’s via the PCIe to Sata controller.

My question is twofold:

  1. How to setup the two HDD’s best, so they are accessible from mac, windows, linux?
    reading up on the internet it seemed best to format as exfat. I tried parted, doesn’t have this function so I followed: How to Format a Drive as exFAT on Linux but now I ended up with two HDD’s which the jetson recognizes as 2tb instead of 4tb each. which I thought was due to partition table, so I followed: Converting between GPT and MBR hard drive without losing data - Super User but still no succes. How should I proceed?

  2. After the steps I did in #1, I suddenly get errors during booting like:

unable to mount /media

PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)

Multiple forum topics here talk about setting the PCIE ASPM to off in a config file. like here: PCIE x4, only 658MB/s - #15 by vidyas

I’ve used the gunzip < /proc/config.gz | egrep ‘PCIEASPM’ command and I see

CONFIG_ PCIEASPM =y

CONFIG_ PCIEASPM _DEBUG is not set

CONFIG_ PCIEASPM _DEFAULT is not set

CONFIG_ PCIEASPM _POWERSAVE=y

CONFIG_ PCIEASPM _PERFORMANCE is not set

So I would like to try and turn off the PCIEASPM power saving. How would I do that? I found this: How to disable the PCIe ASPM on TX1 ? - #2 by vidyas but I don’t know about which config file they are talking? And how to change it? Please ELI5 (Explain to me like i’m five)! I would be greatly thankful.

PS: My overall goal is to create a raid1 and later maybe raid4 with the Sata HDD’s, accessible over the internet. Like a NAS.

Kind regards,

Jeroen

2

Just a thought on this…if you use a method of export which depends on the filesystem being mounted, then you would need exFAT support in the kernel to do the mount. This would be typical for sharing, e.g., network neighborhood, NFS, so on. If instead you export this as a partition, then no mount would be required from the Jetson acting as a server, but this would only be usable by one computer at a time, e.g., via iSCSI.

To support exFAT the kernel would need to have option “CONFIG_EXFAT_FS”. Can you mount your exFAT system? If not, then check this:
zcat /proc/config.gz | egrep -i 'exfat'
…and from what I can tell this only became available on 5.x kernels. If you do not have this, then what is your “uname -r”? I am guessing 4.9.140-tegra.

Sorry, I have not seen any backporting of that driver, nor have I seen anyone running a 5.x kernel.

For each of the drives involved, assuming you fill in the drive name, what do you see on each for:
sudo gdisk -l /dev/sd...whatever...

How did you format the partitions to exFAT?

3

Hi Linuxdev, thank you for your answer!

No I cannot mount exFat currently. As soon as I try to mount my drives in my fstab like that, it won’t boot up anymore.

I tried the zcat command, but including the egrep statement it returns nothing. without the egrep command it gives me the a huge list. I tried searching in that file for the file system parts, I see ext3, ext4, fat, but no exfat. So I guess I don’t have the exfat support. Which I hoped would happen by using “sudo apt-get install exfat-fuse exfat-utils” but apparantly not…

uname -r indeed returns 4.9.140-tegra.

For the gdisk command I get different responses for /dev/sdb and /dev/sdb1, which seems problematic as well?

/dev/sdb

GPT fdisk (gdisk) version 1.0.3

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sdb: 7814037168 sectors, 3.6 TiB
Model: ST4000VN008-2DR1
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 2AF463BB-A7B3-4F68-B5EC-399BED1539D2
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 1539693 sectors (751.8 MiB)
Number Start (sector) End (sector) Size Code Name
1 2048 7812499455 3.6 TiB 8300 primary

and for /dev/sdb1

GPT fdisk (gdisk) version 1.0.3
Partition table scan:
MBR: MBR only
BSD: not present
APM: not present
GPT: not present
***************************>
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory.

Disk /dev/sdb1: 7812497408 sectors, 3.6 TiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): D820B9B4-67D2-4589-86A2-D0813C519B9A
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7812497374
Partitions will be aligned on 2048-sector boundaries
Total free space is 7812497341 sectors (3.6 TiB)

I’m pretty lost right now, as I feel I’m trying to do something very basic, but somehow it’s one big shitshow right now haha. Do you see a chance to get this sort of working? Or should I try to go a totally different route?

4

I think that the kernel (at least this version) does not support exFAT, and thus cannot mount it.

In a case where you want to add an entry to fstab and you do not tell boot that mount failure is allowed, then boot will halt until succeeding in that mount. Catch-22: Boot can’t continue because the kernel cannot mount, and mount cannot continue because the kernel does not know how to.

FYI, in “/etc/fstab”, you will see several “fields”. The fourth field usually has “defaults” in it. “defaults” is equivalent to a comma delimited list of:
rw,suid,dev,exec,auto,nouser,async

If you were to use argument “nofail” in this fourth field, but otherwise be the same as “defaults”, then boot would continue upon failure:
rw,suid,dev,exec,auto,nouser,async,nofail

Having nofail will not allow the kernel to mount a filesystem type it does not understand, but it will allow the unit to continue booting with the failure. Until we get a 5.x series kernel I do not think there is any way to do this without back-porting the driver into the current 4.x series kernel. The lack of the symbol in “/proc/config.gz” illustrates that not only does the kernel not have this configured, but also that the kernel has never seen the possibility of that driver (because the driver did not exist in the 4.9 kernel).

When listing partitions with gdisk, consider that “/dev/sdb” is the disk as a whole, and “/dev/sdb1” is just a single partition. A partition is missing much preamble content which a disk as a whole is not missing. There is nothing wrong with the gdisk output you had.

Basically, I do not see any exFAT filesystem type succeeding without the kernel having the driver, and I do not see any “reasonable” route to getting that exFAT driver without going to a new kernel. There are many dependencies, and so a new kernel is more difficult than it sounds, and probably requires waiting for a release with a 5.x series kernel.

5

Thank you again for taking your time to answer. Basically you’re saying it’s not gonna happen right? Are there other possibilities to make drives shareable between windows,linux,mac? EXT4?

6

Correct, mounting as exFAT is probably not going to happen (at least for several months when an L4T release supports a newer kernel under Ubuntu 20.04).

When you share a drive which is not mounted on the Jetson, then the Jetson won’t care about filesystem type, but those setups are not for sharing among multiple computers. Export of a partition as a drive using this method is restricted to a single remote computer accessing the drive at any time. The drive becomes binary data and the exporting system does not need to understand any filesystem type, but this binary stream cannot be shared by more than one computer at a time any more than a physical hard drive can be installed on two or more computers at the same time.

Any filesystem type which the Jetson understands can be shared among as many computers as you wish through software to export the partition. However, this also implies the remote computer will probably need to have software to deal with the export protocol. This may not be as bad as it sounds in some cases since Linux may have some sort of well-known “adapter” to this, but I do not know what filesystem types a Mac understands (other than HFS).

To illustrate, a partition may be ext4, but exported using Samba. Samba exports files through the Windows netbios protocol. Windows does not need to understand ext4 since netbios does the conversion.

Another illustration is that NFS (there is more than one version of this) exports using an NFS protocol. Anything understanding that protocol can share that filesystem. I don’t think Windows can handle that without purchasing some third party software. Mac might, it is more *NIX and this is the *NIX equivalent to the Windows netbios, but I don’t know for sure.

In all cases where there is an adapter protocol the operating system doing the export (the Jetson) must be able to mount and examine files since this is an export of files rather than an export of the binary data of a partition. An example of exporting as binary data is iSCSI.

7

Disabling ASPM can be achieved in the following ways for a platform. Hope this helps.

  • Disabling from the beginning (i.e. while booting itself)
    • Appending ‘pcie_aspm=off’ to the kernel command line
    • Removing “CONFIG_PCIEASPM_POWERSAVE=y” and setting “CONFIG_PCIEASPM_PERFORMANCE=y” in the kernel configuration
  • Disabling after system boots to console
    • Executing the below command once the system boots to console
      • echo “performance” > /sys/module/pcie_aspm/parameters/policy