JetPack4.3 PCIE virtual ethernet performance issue

53216142 · April 8, 2020, 11:00pm

Hi Guys,

I have download JetPack4.3 and do code sync with ./source_sync.sh -t tegra-l4t-r32.3.1. Then follow this topic to apply those patches.
The bandwidth of of virtual ethernet over PCIe between two xaviers is low

Current git status show below.

 JetPack_4.3_Linux_P2888/Linux_for_Tegra/sources/kernel/nvidia$ git lg
* df1762c - (11 months ago) t19x: mce: rt: enable rt configuration support - Suresh Venkatachalam (HEAD -> mybranch_2020-04-09-1586385193, tag: tegra-l4t-r32.3.1, origin/l4t/l4t-r32.3.1)
* 473478f - (4 months ago) video: tegra: host: nvdla: set EMC rate on device open - Shridhar Rasal
* f75a66d - (4 months ago) drivers:media: camera_common - Anubhav Rai
* 2300b65 - (4 months ago) dt-bindings: carmel PMU: add Carmel uncore PMU drivormatted text by 4 spaces

If I apply patch 0001-PCI-Enable-NAPI-in-T2T-framework.patch, We got follow error message. Can you point me out how to resolve it?

Linux_for_Tegra/sources/kernel/nvidia$ git status
On branch mybranch_2020-04-09-1586385193
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   drivers/net/ethernet/nvidia/pcie/tegra_vnet.c
	modified:   drivers/pci/endpoint/functions/pci-epf-tegra-vnet.c

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	drivers/pci/endpoint/functions/pci-epf-tegra-vnet.c.rej

kayccc · April 16, 2020, 4:24am

Hi 53216142,

The patch was included at the next release, please try it again after we published it at coming weeks. Thanks

bdehj · April 22, 2020, 5:18am

Hi kayccc
The latest JetPack4.4 already include this pcie vnet high performance feature?

kayccc · April 22, 2020, 5:23am

Yes, it should be there.
If still met issue, please open a new topic. Thanks

bdehj · April 22, 2020, 5:28am

after kernel source fetch , I found a new tag l4t/l4t-r32.4.2. Then I switch to l4t/l4t-r32.4.2.
seems git lg doesn’t include high performance patch yet?

Do I need to switch special tag here?

/sources/kernel/nvidia/drivers/pci/endpoint/functions$ git lg .
* 62133a3 - (3 months ago) PCI: tvnet: add tegra_vnet.h - Om Prakash Singh
* 8ac2ca2 - (3 months ago) PCI: tvnet: update function name - Om Prakash Singh
* 55c9f7c - (3 months ago) PCI: add change_mtu callback for tvnet driver - Om Prakash Singh
* 36c404b - (9 months ago) PCI: Process control messages in the same context - Manikanta Maddireddy
* 53d122e - (9 months ago) PCI: Raise interrupt when full buffer is processed - Manikanta Maddireddy
* 1a91302 - (9 months ago) PCI: Optimize read-write counter accesses - Manikanta Maddireddy
* 49bddf9 - (9 months ago) PCI: Enable NAPI in T2T framework - Manikanta Maddireddy

omp · April 22, 2020, 2:32pm

Tag l4t/l4t-r32.4.2 (Jetpack 4.4 developer preview) should include all require patch for getting 5 Gbps.
Please get the performance number with new build.

Thanks,
Om

bdehj · April 24, 2020, 5:57am

Our two xavier use below connection with special cable.
With JetPack4.3, it works well and can do iperf and scp operation with pcie eth1 interface.

After install with JetPack4.4, we get below PCIe Bus Error message. The same setup but got different result.
We have some concern below:

Both JetPack 4.3 and JetPack4.4 use PCIE Gen4, is that right?
our cable still not perfect, we should use nvidia defined special cable. Which cable we should use and where we can buy it?
can we just update dtb to slow up Gen4 to Gen1, maybe the issue will disappear? Bug performance may be down.
Within JetPacke 4.4. we s./source_sync.sh -k l4t/l4t-r32.4.2 to do source download, is that match with JetPack4.4?

1397798761782352111440×1080 146 KB

[0011.768] I> Kernel EP: 0x80080000, DTB: 0x90000000
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.140-tegra (buildbrain@mobile-u64-3193) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) ) #1 SMP PREEMPT Wed Apr 8 18:15:20 PDT 2020
[    0.000000] Boot CPU: AArch64 Processor [4e0f0040]
[    0.000000] OF: fdt:memory scan node memory, reg size 48,
[    0.000000] OF: fdt: - 80000000 ,  2c000000
[    0.000000] OF: fdt: - ac200000 ,  44600000
[    0.000000] OF: fdt: - 100000000 ,  380000000
[    0.000000] earlycon: tegra_comb_uart0 at MMIO32 0x000000000c168000 (options '')
[    0.000000] bootconsole [tegra_comb_uart0] enabled
[    0.000000] Found tegra_fbmem: 00800000@a069c000
[    0.000000] Found lut_mem: 00002008@a0697000
��WARNING: pll_d3 has no dyn ramp
��[    5.404259] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.404499] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[    5.404677] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[    5.406033] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.406264] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[    5.406439] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[    5.408102] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.408336] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[    5.408509] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[    5.410211] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.410445] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[    5.410652] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[    5.416528] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.424548] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000081/0000e000
[    5.432675] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[    5.439493] pcieport 0005:00:00.0:    [ 7] Bad DLLP              
[    5.448099] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.456057] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[    5.464437] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[    5.475437] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.481770] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[    5.490165] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[    5.498548] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.506984] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[    5.515449] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[    5.525918] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.532264] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[    5.540657] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[    5.549087] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.557976] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[    5.566106] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[    5.574324] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[    5.583446] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[    5.591582] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)

bdehj · April 24, 2020, 6:21am

Omp,

Another question, after sources_sync with JetPack4.4, we didn’t found those patch history? Do you have any comments?
Download patches from:
https://drive.google.com/a/nvidia.com/file/d/1OSii9BCiMxxnvvER5T8YZKVElYUfnsOS/view?usp=sharing

omp · April 24, 2020, 6:36am

Hi bdehj,
from your comment you are only checking git history in “kernel/nvidia/drivers/pci/endpoint/functions” folder,
if you check patches carefully most of patches are for kernel/nvidia/drivers/net/ethernet/nvidia/pcie/

and one patch is in hardware/nvidia/soc/t19x/kernel-dts/.

Please point me out which patch is missing in l4t/l4t-r32.4.2 release.

Thanks,
Om

bdehj · April 24, 2020, 7:11am

omp,

about PCIE Bus Error.

I just only download origin JetPack4.4, don’t do build any more.

Can you point me out how to resolve this PCIE Bus Error?

bdehj · April 25, 2020, 12:55am

omp,

I have done some testing below.

burn both EP and RC with original JetPack4.3 packet, and it works for eth1 iperf3.
replace both EP and RC file /boot/Image with JetPack4.4 packet version which is under kernel/Image
do the same testing it works well and can bootup correctly.
then I update both EP and RC dtb with command “sudo ./flash.sh -k kernel-dtb jetson-xavier mmcblk0p1” under JetPacket 4.4
bootup both EP and RC then I see de PCIe Bus Err showing.

Seems some changing in dtb will cause PCIe Bus Err showing, can you reproduce on your side?
What is the key dts changing between JetPack 4.3 and 4.4?

I still found anther two issues:

replace JetPack 4.4 Image version don’t see any improvement on iperf3 tcp testing still 100M band.
if do iperf3 udp testing, there is coredump.

on PCIE RC side

    pcie-rc:~$ sudo ifconfig eth1 192.168.2.1;iperf3 -s
    -----------------------------------------------------------
    Server listening on 5201
    -----------------------------------------------------------
    Accepted connection from 192.168.2.2, port 48820
    [  5] local 192.168.2.1 port 5201 connected to 192.168.2.2 port 48822
    [ ID] Interval           Transfer     Bandwidth
    [  5]   0.00-1.00   sec  8.08 MBytes  67.8 Mbits/sec                  
    [  5]   1.00-2.00   sec  12.0 MBytes   100 Mbits/sec                  
    [  5]   2.00-3.00   sec  12.2 MBytes   102 Mbits/sec                  
    [  5]   3.00-4.00   sec  12.3 MBytes   103 Mbits/sec                  
    [  5]   4.00-5.00   sec  11.6 MBytes  97.1 Mbits/sec                  
    [  5]   5.00-6.00   sec  11.8 MBytes  98.8 Mbits/sec                  
    [  5]   6.00-7.00   sec  12.3 MBytes   103 Mbits/sec                  
    [  5]   7.00-8.00   sec  12.4 MBytes   104 Mbits/sec                  
    [  5]   8.00-9.00   sec  11.4 MBytes  95.3 Mbits/sec                  
    [  5]   9.00-10.00  sec  12.1 MBytes   102 Mbits/sec                  
    [  5]  10.00-10.05  sec   608 KBytes  99.0 Mbits/sec                  
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth
    [  5]   0.00-10.05  sec  0.00 Bytes  0.00 bits/sec                  sender
    [  5]   0.00-10.05  sec   117 MBytes  97.4 Mbits/sec                  receiver
    -----------------------------------------------------------
    Server listening on 5201
    -----------------------------------------------------------
    Accepted connection from 192.168.2.2, port 48824
    [  5] local 192.168.2.1 port 5201 connected to 192.168.2.2 port 48986
    [ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
    [  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)  
    [  139.748695] tegradc 15200000.nvdisplay: blank - powerdown
    [  139.793566] extcon-disp-state external-connection:disp-state: cable 48 state 0
    [  139.793573] Extcon AUX2(HDMI) disable
    [  139.820022] tegra_nvdisp_handle_pd_disable: Powergated Head1 pd
    [  139.822376] tegra_nvdisp_handle_pd_disable: Powergated Head0 pd
    [  139.822893] tegradc 15200000.nvdisplay: unblank
    [  139.839843] tegra_nvdisp_handle_pd_enable: Unpowergated Head0 pd
    [  139.840088] tegra_nvdisp_handle_pd_enable: Unpowergated Head1 pd
    [  139.852734] Parent Clock set for DC plld3
    [  139.857654] tegradc 15200000.nvdisplay: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_111m_223m
    [  139.859367] tegradc 15200000.nvdisplay: hdmi: get RGB quant from EDID.
    [  139.859384] tegradc 15200000.nvdisplay: hdmi: get YCC quant from EDID.
    [  139.901023] extcon-disp-state external-connection:disp-state: cable 48 state 1
    [  139.901032] Extcon AUX2(HDMI) enable
    [  140.495865] skbuff: skb_over_panic: text:ffffff8008938f84 len:8234 put:8234 head:ffffffc3ba04a600 data:ffffffc3ba04a640 tail:0x206a end:0x640 dev:eth1
    [  140.496851] ------------[ cut here ]------------
    [  140.496940] kernel BUG at /home/nvidia/nvidia_sdk/JetPack_4.3_Linux_P2888/Linux_for_Tegra/sources/kernel/kernel-4.9/net/core/skbuff.c:105!
    [  140.496987] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
    [  140.497055] Modules linked in:
    [  140.497330] CPU: 0 PID: 1177 Comm: kworker/0:2 Not tainted 4.9.140+ #2
    [  140.497358] Hardware name: Jetson-AGX (DT)
    [  140.497502] Workqueue: events process_ep2h_msg
    [  140.497547] task: ffffffc3e985aa00 task.stack: ffffffc3e98b8000
    [  140.497653] PC is at skb_panic+0x6c/0x70
    [  140.497709] LR is at skb_panic+0x6c/0x70
    [  140.497767] pc : [<ffffff8008d6e674>] lr : [<ffffff8008d6e674>] pstate: 20c00045
    [  140.497798] sp : ffffffc3e98bbc90
    [  140.497901] x29: ffffffc3e98bbca0 x28: ffffffc3d6730c40 
    [  140.498036] x27: ffffffc3d821a9d0 x26: ffffff8023005018 
    [  140.498134] x25: ffffff8023003000 x24: ffffffc3d821a000 
    [  140.498234] x23: ffffffc3c8b63d00 x22: 000000000000202a 
    [  140.498332] x21: ffffff8008938f84 x20: ffffff8009145af0 
    [  140.498632] x19: ffffffc3c8b63d00 x18: 0000000000000060 
    [  140.498732] x17: 000000000000c5c5 x16: 00000950fffff6af 
    [  140.498826] x15: 00000000fffffffd x14: 6233636666666666 
    [  140.498918] x13: 663a646165682034 x12: 3332383a74757020 
    [  140.499012] x11: 343332383a6e656c x10: 2034386638333938 
    [  140.499195] x9 : 3030386666666666 x8 : ffffffc3ffc318a0 
    [  140.499290] x7 : 0000000000000000 x6 : 0000000016d3d2b6 
    [  140.499378] x5 : 0000000000000000 x4 : 0000000000000000 
    [  140.499473] x3 : ffffffffffffffff x2 : 00000043f645a000 
    [  140.499568] x1 : ffffffc3e985aa00 x0 : 000000000000008a 
    [  140.499592] 
    [  140.499751] Process kworker/0:2 (pid: 1177, stack limit = 0xffffffc3e98b8000)
    [  140.499779] Call trace:
    [  140.499857] [<ffffff8008d6e674>] skb_panic+0x6c/0x70
    [  140.499918] [<ffffff8008d7099c>] skb_put+0x8c/0x90
    [  140.499978] [<ffffff8008938f84>] process_ep2h_msg+0x13c/0x198
    [  140.500046] [<ffffff80080d4e58>] process_one_work+0x1e8/0x490
    [  140.500112] [<ffffff80080d5158>] worker_thread+0x58/0x4c0
    [  140.500436] [<ffffff80080dbb28>] kthread+0xd8/0xf0
    [  140.500501] [<ffffff8008083850>] ret_from_fork+0x10/0x40
    [  140.598416] t19x-arm-smmu 12000000.iommu: Unhandled context fault: smmu0, iova=0xfff8b000, fsynr=0xc0013, cb=3, sid=91(0x5b - PCIE5), pgd=4575eb003, pud=4575eb003, pmd=457618003, pte=0
    [  140.599345] t19x-arm-smmu 12000000.iommu: Unhandled context fault: smmu0, iova=0xfff86000, fsynr=0xc0013, cb=3, sid=91(0x5b - PCIE5), pgd=4575eb003, pud=4575eb003, pmd=457618003, pte=0
    [  140.599979] t19x-arm-smmu 12000000.iommu: Unhandled context fault: smmu0, iova=0xfff66000, fsynr=0xc0013, cb=3, sid=91(0x5b - PCIE5), pgd=4575eb003, pud=4575eb003, pmd=457618003, pte=0
    [  140.600722] t19x-arm-smmu 12000000.iommu: Unhandled context fault: smmu0, iova=0xfff5d000, fsynr=0xc0013, cb=3, sid=91(0x5b - PCIE5), pgd=4575eb003, pud=4575eb003, pmd=457618003, pte=0
    [  140.601399] t19x-arm-smmu 12000000.iommu: Unhandled context fault: smmu0, iova=0xffda1000, fsynr=0xc0013, cb=3, sid=91(0x5b - PCIE5), pgd=4575eb003, pud=4575eb003, pmd=443da2003, pte=0
    [  140.601916] t19x-arm-smmu 12000000.iommu: Unhandled context fault: smmu0, iova=0xffda1000, fsynr=0xc0013, cb=3, sid=91(0x5b - PCIE5), pgd=4575eb003, pud=4575eb003, pmd=443da2003, pte=0
    [  140.603299] mc-err: vpr base=0:0, size=0, ctrl=3, override:(a01a8340, fcee10c1, 1, 0)
    [  140.603370] mc-err: (255) csw_pcie5w: MC request violates VPR requirements
    [  140.603459] mc-err:   status = 0x0ff740e3; addr = 0xffffffff00; hi_adr_reg=008
    [  140.603491] mc-err:   secure: yes, access-type: write
    [  140.603855] mc-err: mcerr: unknown intr source intstatus = 0x00000000, intstatus_1 = 0x00000000
    [  140.604783] t19x-arm-smmu 12000000.iommu: Unhandled context fault: smmu0, iova=0xffd84000, fsynr=0xc0013, cb=3, sid=91(0x5b - PCIE5), pgd=4575eb003, pud=4575eb003, pmd=443da2003, pte=0
    [  140.605231] t19x-arm-smmu 12000000.iommu: Unhandled context fault: smmu0, iova=0xffd62000, fsynr=0xc0013, cb=3, sid=91(0x5b - PCIE5), pgd=4575eb003, pud=4575eb003, pmd=443da2003, pte=0
    [  140.606317] mc-err: vpr base=0:0, size=0, ctrl=3, override:(a01a8340, fcee10c1, 1, 0)
    [  140.606421] mc-err: (255) csw_pcie5w: MC request violates VPR requirements
    [  140.606476] mc-err:   status = 0x0ff740e3; addr = 0xffffffff00; hi_adr_reg=008
    [  140.606505] mc-err:   secure: yes, access-type: write
    [  140.606592] mc-err: mcerr: unknown intr source intstatus = 0x00000000, intstatus_1 = 0x00000000
    [  140.610315] t19x-arm-smmu 12000000.iommu: Unhandled context fault: smmu0, iova=0xffd58000, fsynr=0xc0013, cb=3, sid=91(0x5b - PCIE5), pgd=4575eb003, pud=4575eb003, pmd=443da2003, pte=0
    [  140.610807] t19x-arm-smmu 12000000.iommu: Unhandled context fault: smmu0, iova=0xffd54000, fsynr=0xc0013, cb=3, sid=91(0x5b - PCIE5), pgd=4575eb003, pud=4575eb003, pmd=443da2003, pte=0
    [  140.614934] mc-err: Too many MC errors; throttling prints
    [  141.518931] ---[ end trace 01525e739c6eb4e8 ]---
    [  141.544974] Kernel panic - not syncing: Fatal exception in interrupt
    [  141.544980] SMP: stopping secondary CPUs
    [  141.544999] Kernel Offset: disabled
    [  141.545001] Memory Limit: none
    [  142.014180] trusty-log panic notifier - trusty version Built: 22:43:40 Dec  9 2019 
    [  142.014180] Rebooting in 5 seconds..

on PCIE EP side

    pcie-ep:~$ sudo ifconfig eth1 192.168.2.2;iperf3 -c 192.168.2.1 -b 1000M
    Connecting to host 192.168.2.1, port 5201
    [  4] local 192.168.2.2 port 48822 connected to 192.168.2.1 port 5201
    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
    [  4]   0.00-1.00   sec  9.52 MBytes  79.9 Mbits/sec    0    376 KBytes       
    [  4]   1.00-2.00   sec  12.1 MBytes   101 Mbits/sec    0    510 KBytes       
    [  4]   2.00-3.00   sec  12.5 MBytes   105 Mbits/sec    0    529 KBytes       
    [  4]   3.00-4.00   sec  12.1 MBytes   102 Mbits/sec    0    556 KBytes       
    [  4]   4.00-5.00   sec  11.9 MBytes  99.8 Mbits/sec    0    556 KBytes       
    [  4]   5.00-6.00   sec  11.6 MBytes  97.5 Mbits/sec    0    585 KBytes       
    [  4]   6.00-7.00   sec  12.4 MBytes   104 Mbits/sec    0    585 KBytes       
    [  4]   7.00-8.00   sec  12.1 MBytes   102 Mbits/sec    0    585 KBytes       
    [  4]   8.00-9.00   sec  11.6 MBytes  97.5 Mbits/sec    0    585 KBytes       
    [  4]   9.00-10.00  sec  12.4 MBytes   104 Mbits/sec    0    585 KBytes       
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth       Retr
    [  4]   0.00-10.00  sec   118 MBytes  99.2 Mbits/sec    0             sender
    [  4]   0.00-10.00  sec   117 MBytes  97.9 Mbits/sec                  receiver

    iperf Done.
    pcie-ep:~$ sudo ifconfig eth1 192.168.2.2;iperf3 -u -c 192.168.2.1 -b 1000M
    Connecting to host 192.168.2.1, port 5201
    [  4] local 192.168.2.2 port 48986 connected to 192.168.2.1 port 5201

bdehj · May 12, 2020, 12:24am

omp, do you have any comments about pcie bus error issue?

omp · May 13, 2020, 7:42am

PCIe bus error is likely due to cable that you are using.

You will need E3317 like TX/RX swap board.

Please refer to below document for detail specification about E3317 board:

Jetson_AGX_Xavier_PCIe_Endpoint_Design_Guidelines.pdf (1.9 MB)

Also go through this thread:

Thanks,
Om

Andrey1984 · May 29, 2021, 9:45am

@omp
do we need to disable y->n
in

CONFIG_PCIEASPM_POWERSAVE=y

in order to address the error

pcieport 0005:00:00.0 AER: Multiple Corrected error recieved:  id=0000
PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
0005:00:00.0 :   device [10de:1ad0] error status/mask=00000001/0000e000
pcieport 0005:00:00.0 :    [ 0] Receiver Error         (First)

can it be done withut rebuilding the kernel from sources?

omp · May 31, 2021, 7:53am

I believe ASPM is already disabled for C5 controller. you can check ASPM status using “lspci -s 0005:00:00.0 -vvv | grep ASPM” command

Andrey1984 · May 31, 2021, 8:59am

root@agx:/home/nvidia# /proc/cmdline
bash: /proc/cmdline: Permission denied
root@agx:/home/nvidia# lspci -s 0005:00:00.0 -vvv | grep ASPM
root@agx:/home/nvidia# cat /proc/config.gz | gunzip |grep CONFIG_PCIEASPM_POWERSAVE
CONFIG_PCIEASPM_POWERSAVE=y
root@agx:/home/nvidia#