NVME external drive not recognized on PCIe C0

I am trying to get my Orin SoMs to recognize an external NVME 980 GB drive and it is set up as PCIe Controller C0. I believe my pinmux is correct.

My ODMDATA (below) is set for UPHY Configuration #2.
https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/HR/JetsonModuleAdaptationAndBringUp/JetsonAgxOrinSeries.html#uphy-lane-configuration

ODMDATA=“gbe-uphy-config-0,nvhs-uphy-config-1,hsio-uphy-config-16,hsstp-lane-map-3”;

I changed the appropriate files from the Developers Guide here:

https://docs.nvidia.com/jetson/archives/r35.2.1/DeveloperGuide/text/HR/JetsonModuleAdaptationAndBringUp/JetsonAgxOrinSeries.html#enable-pcie-in-a-customer-cvb-design

I changed the code for PCIe x1 (C0) and PCIe x8 (C7) in the files: tegra234-p3737-pcie.dtsi, tegra234-mb1-bct-pinmux-p3701-0000-a04.dtsi, & tegra234-mb1-bct-gpio-p3701-0000-a04.dtsi. I recompiled the kernel, copies files over, and flashed.

After booting without issue and logging into the SoM, running the lsblk command does not show the NVME drive present.

Is there a step that I am missing?

Share the file you have modified.

Share the dmesg.

@WayneWWW
Attached are the three files I modified according to the dev guide for C0 & C7.

tegra234-mb1-bct-gpio-p3701-0000-a04 1.dtsi.txt (4.7 KB)
tegra234-mb1-bct-pinmux-p3701-0000-a04.dtsi.log (63.6 KB)
tegra234-p3737-pcie.dtsi.log (2.0 KB)

Also attached is the dmesg.log
dmesg.log (68.6 KB)

please share your flash log too.

1 Like

Here is the flash log.

log_flash.txt (625.2 KB)

Thank you!

please use dtc tool to convert this file back to dts and share the result.

tegra234-bpmp-3701-0005-3737-0000.dtb

The converted is attached (with the .txt added so it could be uploaded):

tegra234-bpmp-3701-0005-3737-0000.dts.txt (203.4 KB)

Where did you copy this file out?

The uphy setting is still the old one. Not updated with your ODMDATA.

All the files are from the same Linux_for_tegra directory that was used to flash (with the included flash.log earlier). The converted dts file was from the /bootloader directory from that Linxu_for_tegra directory.

What should I be looking for in the tegra234-bpmp-3701-0005-3737-0000.dts file? That file I didn’t change (I dont think).

Open the file you shared and search “uphy” and it will show.

You are right, that converted dts file does not show my current ODMDATA values.

How is that file created? I am using a custom *.conf that has the ODMDATA values as:
ODMDATA=“gbe-uphy-config-0,nvhs-uphy-config-1,hsio-uphy-config-16,hsstp-lane-map-3”;

My custom conf is also including the p3701.conf.common file, which has the original incorrect ODMDATA values that I’m seeing in the tegra234-bpmp-3701-0005-3737-0000.dtb.txt file. I thought my custom .conf would supersede the p3701.conf.common values.

But just to be sure, I commented out that ODMDATA line in the p3701.conf.common to make sure it is grabbing my modified ODMDATA, but after flashing, I still see the same incorrect results in the tegra234-bpmp-3701-0005-3737-0000.dts file.

Where is the tegra234-bpmp-3701-0005-3737-0000.dtb file getting the ODMDATA from? Do I need to change that ODMDATA it in another place other than my .conf?

Just to recap-
My flash command is:
sudo ./flash.sh auto2-gpgpu mmcblk0p1

My auto2-gpgpu.conf file contains the modified ODMDATA=“gbe-uphy-config-0,nvhs-uphy-config-1,hsio-uphy-config-16,hsstp-lane-map-3”;

Thanks.

You can also just put this uphy setting in the bpmp dtb to make sure it would work.

If NVMe is still not able to get detected in C0 after this, try other NVMe to test too.

Ok, I will try that.
Do I need to change the settings in the tegra234-bpmp-3701-0005-3737-0000.dts file and then convert it back to .dtb? How is this done?

Yes, use dtc tool to convert it from binary to source and again source to binary.

1 Like

So I made the change to the tegra234-bpmp-3701-0005-3737-0000.dts.txt then converted it back to a .dtb using the dtc tool, to match ’ ODMDATA=“gbe-uphy-config-0,nvhs-uphy-config-1,hsio-uphy-config-16,hsstp-lane-map-3”; ’ as

    uphy {
            status = "okay";
            hsio-uphy-config = <0x0>;
            hsstp-lane-map = <0x3>;
            nvhs-uphy-config = <0x1>;
            gbe-uphy-config = <0x10>;
    };

But now it does not boot after flashing. This is what I see during boot, then stops:

��NOTICE: BL31: v2.6(release):346877e39
NOTICE: BL31: Built : 12:32:40, Aug 1 2023
I/TC: Physical secure memory base 0x103c040000 size 0x3fc0000
��DCE: FW Boot Done
��I/TC:
I/TC: Non-secure external DT found
I/TC: OP-TEE version: 3.21 (gcc version 9.3.0 (Buildroot 2020.08)) #2 Tue Aug 1 19:39:55 UTC 4
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidell
I/TC: Primary CPU initializing
I/TC: Test OEM keys are being used. This is insecure for shipping products!
I/TC: Primary CPU switching to normal world boot
��
Jetson UEFI firmware (version 4.1-33958178 built on 2023-08-01T19:34:02+00:00)

��I/TC: Reserved shared memory is disabled
I/TC: Dynamic shared memory is enabled
I/TC: Normal World virtualization support is disabled
I/TC: Asynchronous notifications are disabled
��

��E/TC:?? 00 get_rpc_alloc_res:645 RPC allocation failed. Non-secure world result: ret=0xfff0
E/LD: init_elf:486 sys_open_ta_bin(bc50d971-d4c9-42c4-82cb-343fb7f37896)
E/TC:?? 00 ldelf_init_with_ldelf:131 ldelf failed with res: 0xffff000c
��

��Unhandled Exception in EL3.
x30 = 0x0000000050000d00
x0 = 0x0000000000000000
x1 = 0x00000000be000011
x2 = 0x0000000000000000
x3 = 0x0000000000000011
x4 = 0x0000000000100000
x5 = 0x000000102d1fe4e8
x6 = 0x0000003c01000000
x7 = 0x0000003c01000000
x8 = 0x00000000000000ff
x9 = 0x000000005001c380
x10 = 0x55aaa055071dbd35
x11 = 0x55aa8255ce1abfe1
x12 = 0x0a0341d0000c0102
x13 = 0x0004ff7f00000000
x14 = 0x00000010065bdba8
x15 = 0x00000010065bdb10
x16 = 0x000000102902803c
x17 = 0x00000000307cf10e
x18 = 0x0000001028f3b2f0
x19 = 0x000000005001cec0
x20 = 0x0000000000000000
x21 = 0x0000000000000000
x22 = 0x0000000000000000
x23 = 0x0000000000000000
x24 = 0x0000000000000000
x25 = 0x0000000000000000
x26 = 0x0000000000000000
x27 = 0x0000000000000000
x28 = 0x0000000000000000
x29 = 0x0000000000000000
scr_el3 = 0x000000000003073d
sctlr_el3 = 0x00000000b0cd183f
cptr_el3 = 0x0000000000000000
tcr_el3 = 0x0000000080823518
daif = 0x00000000000002c0
mair_el3 = 0x00000000004404ff
spsr_el3 = 0x00000000600003c9
elr_el3 = 0x0000001028f35280
ttbr0_el3 = 0x0000000050026ac1
esr_el3 = 0x00000000be000011
far_el3 = 0x0000000000000000
spsr_el1 = 0x0000000000000000
elr_el1 = 0x0000000000000000
spsr_abt = 0x0000000000000000
spsr_und = 0x0000000000000000
spsr_irq = 0x0000000000000000
spsr_fiq = 0x0000000000000000
sctlr_el1 = 0x0000000030d00800
actlr_el1 = 0x0000000000000000
cpacr_el1 = 0x0000000000300000
csselr_el1 = 0x0000000000000000
sp_el1 = 0x0000000000000000
esr_el1 = 0x0000000000000000
ttbr0_el1 = 0x0000000000000000
ttbr1_el1 = 0x0000000000000000
mair_el1 = 0x0000000000000000
amair_el1 = 0x0000000000000000
tcr_el1 = 0x0000000000000000
tpidr_el1 = 0x0000000000000000
tpidr_el0 = 0x0000000080000000
tpidrro_el0 = 0x0000000000000000
par_el1 = 0x0000000000000800
mpidr_el1 = 0x0000000081000000
afsr0_el1 = 0x0000000000000000
afsr1_el1 = 0x0000000000000000
contextidr_el1 = 0x0000000000000000
vbar_el1 = 0x0000000000000000
cntp_ctl_el0 = 0x0000000000000005
cntp_cval_el0 = 0x00000000108b96a4
cntv_ctl_el0 = 0x0000000000000000
cntv_cval_el0 = 0x0000000000000000
cntkctl_el1 = 0x0000000000000000
sp_el0 = 0x0000001028f3b2f0
isr_el1 = 0x0000000000000040
cpuectlr_el1 = 0xa000000b40543000
gicd_ispendr regs (Offsets 0x200 - 0x278)
Offset: value
0000000000000200: 0x0000000000000000
0000000000000204: 0x0000000000000000
0000000000000208: 0x0000000000000000
000000000000020c: 0x0000000000000000
0000000000000210: 0x0000000000000000
0000000000000214: 0x0000000000000000
0000000000000218: 0x0000000000010000
000000000000021c: 0x0000000000020000
0000000000000220: 0x0000000000000000
0000000000000224: 0x0000000000000000
0000000000000228: 0x0000000000000000
000000000000022c: 0x0000000000000000
0000000000000230: 0x0000000000000000
0000000000000234: 0x0000000000000000
0000000000000238: 0x0000000000000000
000000000000023c: 0x0000000000000000
0000000000000240: 0x0000000000000000
0000000000000244: 0x0000000000000000
0000000000000248: 0x0000000000000000
000000000000024c: 0x0000000000000000
0000000000000250: 0x0000000000000000
0000000000000254: 0x0000000000000000
0000000000000258: 0x0000000000000000
000000000000025c: 0x0000000000000000
0000000000000260: 0x0000000000000000
0000000000000264: 0x0000000000000000
0000000000000268: 0x0000000000000000
000000000000026c: 0x0000000000000000
0000000000000270: 0x0000000000000000
0000000000000274: 0x0000000000000000
0000000000000278: 0x0000000000000000
000000000000027c: 0x0000000000000000

This is actually the correct behavior. When your device tree does not match to your ODMDATA, then it will hit error.

Your previous behavior was wrong. Current behavior is correct.

Please update the device tree then.

I’m not sure what else to change.

I changed the code for PCIe x1 (C0) and PCIe x8 (C7) in the files: tegra234-p3737-pcie.dtsi, tegra234-mb1-bct-pinmux-p3701-0000-a04.dtsi, & tegra234-mb1-bct-gpio-p3701-0000-a04.dtsi . I recompiled the kernel, the device tree and copied over the files.

What else I need to change to match this ODMDATA configuration?

Sorry, I just realized. @jmumper2 didn’t modify bpmp dtb correctly.

Why does this even go wrong with just writing a string into the file?

Jmmper2 changed the bpmp in the converted text format to:

uphy {
        status = "okay";
        hsio-uphy-config = <0x0>;
        hsstp-lane-map = <0x3>;
        nvhs-uphy-config = <0x1>;
        gbe-uphy-config = <0x10>;
}

which matches ODMDATA=“gbe-uphy-config-0,nvhs-uphy-config-1,hsio-uphy-config-16,hsstp-lane-map-3”;, correct?

Then changed the dts text file back to the dtb file via the dtc tool. And flashed. The above error message is what we are currently experiencing…

Wait. I just realized the file could be wrontg. Should the file be?

uphy {
status = “okay”;
hsio-uphy-config = <0x10>;
hsstp-lane-map = <0x3>;
nvhs-uphy-config = <0x1>;
gbe-uphy-config = <0x0>;
}