JetPack 4.6
JetPack5.1
Please test on rel-5.1.4 and run lspci with sudo to provide full information.
BARs should be defined by the PCIe device. Not Jetson side.
Why is the number of BARs (Base Address Registers) of the same PCIe NVMe SSD device different between JetPack 4.6 and JetPack 5.10?
Does that happen to other NVMe SSD as well or only this kind?
or change the question into another way. What is the exact reason that bothers you to concern this BAR number?
The performance of other SSDs remains the same. The main reason I’m concerned about is that I found the read - write performance of the SSD in JetPack 4.6 is about twice that of JetPack 5.1. Through my preliminary investigation, I noticed that the number of BARs is different. The amount of BARs in 4.6 is 64K more than that in 5.1. At present, I think this 64K difference is the cause of the performance issue.
When the types of SSDs are different, the number of BARs is also different.
As other kind of PCIe SSD would not hit this issue, I would suggest you can check with SSD vendor first. Maybe they know what configuration to check first.
Basically if other SSD still have same BARs setting between jp4 and jp5, then it seems not a general problem to jetson pcie driver.
You could also compare the full “lspci -vvv” result between jp4 and jp5 with that SSD but not just that BARs.
Can changes in the Tegra194-pcie driver lead to different BAR settings for the same SSD?
Actually I don’t think that is the cause.
Do you mean that different SSDs result in different numbers of BARs?
Actually the number of BARs should not affect the read/write speed of a NVMe SSD. The issue could be from something else.
That is why better checking with NVMe vendor if they ever tested this SSD on kernel 5.10 and if any known issue.
You didn’t reveal full PCIe state by using lspci -vvv.
This is not needed… I only asked for the result of “sudo lspci -vvv” on jp4 and jp5…
Not sure why you want to share such dummy log…
Why does the performance of an SSD tested with the ‘dd’ command differ between JetPack 4.6 and JetPack 5.1?
請問如果用中文溝通你會比較清楚嗎? 感覺現在對話一直在繞著轉…
ok… 我大概說明一下
所有事情其實前面已經講過了.
-
請你了解一下…我們不是什麼問題都會知道答案. 請你確認一下你現在是單一SSD碰上performance的問題 還是你那邊其他SSD都有這個問題… 現在的狀況就是你丟了一個我們也不知道的SSD出來問我為什麼速度測出來有差. 說實話我也沒辦法回答你.
-
lspci -vvv完整的兩個版本的結果請給出來
I want to point out that the PCI drivers are for the bus itself. The kernel major release changes between Ubuntu 20.04 and Ubuntu 22.04, and the device driver (which sits on a PCI bus) would have perhaps changed between the 4.x kernel and 5.x kernel, or even among significantly different minor releases of one major release. My point being that the drivers to the PCI bus itself might not have anything to do with it, and it might be the driver for the device which is changing. If the device driver changes, then how it reports a given device could perhaps be why lspci changes.
I don’t really know, but you should check exactly which PCI driver release version is used between the two. Once you do that, then check if the driver is loading with different arguments. The very same driver, is passed different arguments or if inheriting a different environment, could act quite differently even if the driver itself is the same.
jp4.5
sudo lspci -vvv -s 0000:01:00.0
0000:01:00.0 Non-Volatile memory controller: Device 1bc0:5236 (rev 01) (prog-if 02 [NVM Express])
Subsystem: Device 1bc0:5236
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 33
Region 0: Memory at 1b40030000 (64-bit, non-prefetchable) [size=16K]
Region 4: Memory at 1b40020000 (64-bit, non-prefetchable) [size=64K]
[virtual] Expansion ROM at 1b40000000 [disabled] [size=128K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 16GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [b0] MSI-X: Enable+ Count=66 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [158 v1] #19
Capabilities: [178 v1] #26
Capabilities: [19c v1] #27
Capabilities: [1f4 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1fc v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=40us
Capabilities: [20c v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [244 v1] #25
Kernel driver in use: nvme
jp5.1
0000:01:00.0 Non-Volatile memory controller: Device 1bc0:5236 (rev 01) (prog-if 02 [NVM Express])
Subsystem: Device 1bc0:5236
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 24
Region 0: Memory at 1b40030000 (64-bit, non-prefetchable) [size=16K]
Region 4: Memory at 1b40020000 (64-bit, non-prefetchable) [size=64K]
Expansion ROM at 1b40000000 [virtual] [disabled] [size=128K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 16GT/s (ok), Width x4 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+
10BitTagComp+, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-, TPHComp-, ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [b0] MSI-X: Enable+ Count=66 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [158 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
LaneErrStat: 0
Capabilities: [178 v1] Physical Layer 16.0 GT/s <?>
Capabilities: [19c v1] Lane Margining at the Receiver <?>
Capabilities: [1f4 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1fc v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Capabilities: [20c v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [244 v1] Data Link Feature <?>
Kernel driver in use: nvme