I’d like to order a x4 Gen4 NVME SSD for an AGX Orin devkit.
Would someone tell about such a model know to work fine at Gen4 for long time ?
Thanks for any feedback
What kind of speed you are getting on the 980 pro, the gen 4 NVME I put in is rated for 7GB/s but I seem capped at around 1.7GB/s.
I bought this drive https://www.amazon.com/Nextorage-Internal-Expansion-NEM-PA1TB-SYM/dp/B09RMHC8L8/
Works fine no issues, I don’t think I have ever seen any kind of storage comparison for the Orin since its very new but it was more in line speed wise with what gen 3 would be expected to do.
Hey richg,
I used the built in ubuntu disk benchmark. I got about 4.1 GB/s. A bit low but still not bad. I wonder if there are better benchmarking tools. I would say this is close to maximum transfer rate for the ARM A78AE. Maybe an NVIDIA engineer knows with their internal testing but my links look to match the spec for PCIe gen4.
If you are having issues check if your pcie links are gen4.
Thanks @RDobitz for sharing this. Looks promising ;-)
Indeed, LnkCap shows that your device is detected as Gen4-able, and furthermore LnkSta just below shows that is indeed currently running in Gen4 mode.
Never seen it throttling back to Gen3 under heavy device usage ?
I have no experience with such Gen4 SSD device, but from what I’ve read it may draw and dissipate about 7W, and throttle back to lower if too hot, so my concern (I currently have no other platform for a Gen4 SSD in case it doesn’t work out).
@richg, this may provide an hypothesis for what you’re seeing. You may check with:
# install nvme-cli if not yet done:
sudo apt install nvme-cli
# Now you may check if supported with:
sudo nvme smart-log /dev/nvme0 | grep 'temperature'
1 Like
good thinking I didn’t think of checking that… That’s not slow, this drive was advertised as having 7000R/6000W and I’m getting 1700. It was advertised as being designed for PS5, Mac, & PC, but in general they were targeting the PS5 people. Do you know of anything else which is needed for enablement? I’ll do some digging, but afaik it was just in the linux kernel, but not sure. Worst case I’ll swap it out.
It was outfitted with a heatsink, and usually when these things overheat you would see it go faster then drop drastically, it seems like its working fine it just can’t break through that 1.7gb/s, I tried bumping the values up in the benchmark settings in disk it only made it more consistently stay at that level.
That’s weird, I tested it again and now it is faster… hmmm curious.
Subsystem: Sony Corporation Device 9122
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 52
Region 0: Memory at 2428000000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 16GT/s (ok), Width x4 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+
10BitTagComp+, 10BitTagReq-, OBFF Not Supported, ExtFmt+, EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-, TPHComp-, ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [d0] MSI-X: Enable+ Count=33 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [e0] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [f8] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [110 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=220us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1e0 v1] Data Link Feature <?>
Capabilities: [200 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [290 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [2a0 v1] Power Budgeting <?>
Capabilities: [300 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
LaneErrStat: 0
Capabilities: [340 v1] Physical Layer 16.0 GT/s <?>
Capabilities: [378 v1] Lane Margining at the Receiver <?>
Kernel driver in use: nvme
Still seems like it should be faster… I expect the 980 Pro is ideal but is there any other drives people have used and found to perform at over 4GB/s (as is that is too slow, well the faster it is the quicker you can load large models or use more performant swap.
Looks like throttling back…and forward.
Check what your device is detected to be able with:
sudo lspci -vv | grep -i "NVM" -A40 | grep -A1 LnkCap
should show 16 GT/s for Gen4
Check what speed it is currently running with:
sudo lspci -vv | grep -i "NVM" -A40 | grep LnkSta:
Also check for errors if any with:
sudo lspci -vv | grep -i "NVM" -A40 | grep AERCap
I’ve accepted @RDobitz’s advice for Samsung 980 pro as it clearly seems to be the best choice so far, but I’d suggest:
- That NVIDIA that has probably tried many devices gives a feedback of Gen4 tests.
- That this topic should not be closed, so that next users can post their findings about other devices (may be cloned to an Orin FAQ)
Another device which should prove very performant on the Jetson Orin is the SN850 based on these findings
Just curious, are you using a M.2 adapter card and installing in the CEM slot?
Standard M.2 80mm PCIe x4 gen 4 onboard… Directly accessible on the bottom.
I was wondering if anyone tried a 8TB nvme on the Orin Agx? Any good ones that could be used AGX? Performance wise how does this hold?