970 PRO NVMe M.2 1TB SSD is not detected on Xavier dev kit

Hi,

I’m having trouble getting Jetson Xavier AGX to detect a connected SSD.
There is no block device file for the SSD present in /dev. lsblk and lspci do not list the device either.

The SSD works and is detect normally on other computers but not on Xavier.
I’ve tested also with 2 other 970 Pro’s but none of them are recognized.

Dmesg does not seem to contain any relevant information but I attached it just in case.

dmesg.txt (68.2 KB)

Any help for fixing this would be appreciated.

Hi,

Is this one 970 EVO NVMe M.2 SSD?

Are you working on Xavier devkit or custom carrier board?

Is this one 970 EVO NVMe M.2 SSD?

No, it’s the PRO model MZ-V7P1T0

Are you working on Xavier devkit or custom carrier board?

Devkit

Is the SSD here connected to M.2 Key-M slot or CEM slot i.e. x16 slot?
If it is connected to M.2 Key-M slot, then make the below modifications in pcie@14180000 DT node or else in pcie@141a0000 node.

  1. Try setting maximum speed to Gen-1 by setting “nvidia,max-speed = <1>”
  2. Try limiting the link width to only x1 by setting num-lanes = <1>;
  3. Try adding nvidia,disable-clock-request;

what is the output of the commands below?

lsblk
lspci

Should be connected to the Key-M slot. Will try changing the settings after I figure out how the device-tree system works…

$ lspci
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 (rev 13)

lsblk.txt (2.2 KB)

is it a default stock kernel? non-modified? modified?
flashed by sdkmanager with Jetpack 4.2? 4.3? DP_4.4?
what is the output of the command below?

 gunzip -c /proc/config.gz |grep NVME

Default stock kernel. Don’t know which JetPack version it was because it was not installed by me.
/etc/nv_tegra_release shows: R32 (release), REVISION: 3.1

   CONFIG_NVME_CORE=y
   CONFIG_BLK_DEV_NVME=y
   # CONFIG_BLK_DEV_NVME_SCSI is not set
   # CONFIG_NVME_TARGET is not set
   # CONFIG_NVMEM is not set

do you consider reinstalling the OS to see if it changes the situation?
Maybe if you install with Jetpack DP_4.4 the situation will be different

I’ll check if that works.

Thanks @Andrey1984 for helping out here. Just want to clarify here issue is about PCIe EP detection. it should work on all Jetpack release. so no need to move to Jetpack 4.4 for this issue

@joel.polso Please try what @vidyas has suggested first.
you mentioned that this nvme card is working on some other host, can you share output of “sudo lspci -vvv”

lspci -vvv output from Jetson:
lspci_vvv.txt (6.6 KB)

lspci -vvv output from an other computer with the same SSD’s, I omitted other devices from this:
lspci_vvv_other.txt (4.4 KB)

@joel.polso Please try below patch.

folder: $TOP/hardware/nvidia/soc/t19x/kernel-dts/

diff --git a/kernel-dts/tegra194-soc/tegra194-soc-pcie.dtsi b/kernel-dts/tegra194-soc/tegra194-soc-pcie.dtsi
index fddb0fb..a100062 100644
--- a/kernel-dts/tegra194-soc/tegra194-soc-pcie.dtsi
+++ b/kernel-dts/tegra194-soc/tegra194-soc-pcie.dtsi
@@ -334,11 +334,11 @@
            interrupts = <0 72 0x04>;       /* controller interrupt */
            interrupt-names = "intr";

-               nvidia,max-speed = <4>;
+               nvidia,max-speed = <3>;
            nvidia,bar0-size = <0x100000>;  /* 1 MB */
            nvidia,device-id = /bits/ 16 <0x1AD4>;
            nvidia,controller-id = <&bpmp 0x0>;
-               num-lanes = <8>;
+               num-lanes = <4>;
            nvidia,aux-clk-freq = <0x13>;
            nvidia,aspm-cmrt = <0x3C>;
            nvidia,aspm-pwr-on-t = <0x14>;

Hi,

I took over the testing from @joel.polso.

Tried recompiling kernel with your patch, but it does not seem to make any difference. the NVMe -drive is still missing.

After booting the system, it says “DTS File Name: path/to/Linux_for_Tegra/sources/kernel/kernel-4.9/arch/arm64/boot/dts/…/…/…/…/…/…/hardware/nvidia/platform/t19x/galen/kernel-dts/common/tegra194-p2888-0001-p2822-0000-common.dtsi”

after digging into the tegra194-p2888-0001-p2822-0000-common.dtsi -file, it seems, that the tegra194-soc-pcie.dtsi, referred by your patch, is not included at all? Am I doing something wrong?

nv_tegra_release is:

R32 (release), REVISION: 4.2, GCID: 20074772, BOARD: t186ref, EABI: aarch64, DATE: Thu Apr 9 01:26:40 UTC 2020

-Jukka

after digging into the tegra194-p2888-0001-p2822-0000-common.dtsi -file, it seems, that the tegra194-soc-pcie.dtsi, referred by your patch, is not included at all? Am I doing something wrong?

tegra194-p2888-0001-p2822-0000-common.dtsi is platform dt file. all module dt will be included in it.

Can you provide me output of below commands:
xxd /proc/device-tree/pcie@14180000/nvidia,max-speed
xxd /proc/device-tree/pcie@14180000/num-lanes

default values are 4 and 8 respectively. If you are flashing it correctly value should be 3 and 4.

make sure after compilation your are updating dt binary before flash

cp $TOP/out/l4t-t186ref-release-aarch64/nvidia/kernel/arch/arm64/boot/dts/tegra194-p2888-0001-p2822-0000.dtb $TOP/out/l4t-t186ref-release-aarch64/full_linux_for_tegra/Linux_for_Tegra/kernel/dtb/

tried to copy the dt binary before flashing but xxd still reports default values:

xxd /proc/device-tree/pcie@14180000/nvidia,max-speed
00000000: 0000 0004 …

xxd /proc/device-tree/pcie@14180000/num-lanes
00000000: 0000 0008 …

Maybe I’m using wrong flash.sh command syntax?
“flash.sh jetson-xavier mmcblk0p1”

Also, should I run the “apply_binaries.sh” before flashing?

Also noticed that the tegra194-p2888-0001-p2822-0000.dtb is copied as I did the kernel compile as described on this document: https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html

Flash command looks fine to me. after compilation (tmp kernel) please check timestamp of mentioned dtb file if it is getting updated.

you can also verify if your change is preset in dtb by decompressing it using below command. to go pcie_ep@14180000 and check the max_speed property

./out/l4t-t186ref-release-aarch64/nvidia/kernel/scripts/dtc/dtc -I dtb -O dts -o temp.txt $TOP/out/l4t-t186ref-release-aarch64/nvidia/kernel/arch/arm64/boot/dts/tegra194-p2888-0001-p2822-0000.dtb

Did the decompress for the tegra194-p2888-0001-p2822-0000.dtb.

Even copied the file from /boot/tegra194-p2888-0001-p2822-0000.dtb at the running Xavier system and checked it too:

    pcie_ep@14180000 {
    ...
            nvidia,max-speed = <0x3>;
            num-lanes = <0x4>;
   ...
  };

seems ok to me, but for some reason xxd still reports:
xxd /proc/device-tree/pcie@14180000/nvidia,max-speed
00000000: 0000 0004
xxd /proc/device-tree/pcie@14180000/num-lanes
00000000: 0000 0008 …

(edit!)

actually I was looking at the wrong place

xxd /proc/device-tree/pcie_ep@14180000/nvidia,max-speed
00000000: 0000 0003 …
xxd /proc/device-tree/pcie_ep@14180000/num-lanes
00000000: 0000 0004 …

seems to be ok.
Still no sign of the NVMe drive though.

Please notice there are two different nodes
pcie_ep@14180000 and pcie@14180000
You need to make change in pcie@14180000