Advice for a x4 Gen4 NVME SSD for AGX Orin devkit

I’d like to order a x4 Gen4 NVME SSD for an AGX Orin devkit.

Would someone tell about such a model know to work fine at Gen4 for long time ?

Thanks for any feedback

I have this and it works great. It comes in many sizes!

1 Like

What kind of speed you are getting on the 980 pro, the gen 4 NVME I put in is rated for 7GB/s but I seem capped at around 1.7GB/s.

I bought this drive https://www.amazon.com/Nextorage-Internal-Expansion-NEM-PA1TB-SYM/dp/B09RMHC8L8/

Works fine no issues, I don’t think I have ever seen any kind of storage comparison for the Orin since its very new but it was more in line speed wise with what gen 3 would be expected to do.

Hey richg,

I used the built in ubuntu disk benchmark. I got about 4.1 GB/s. A bit low but still not bad. I wonder if there are better benchmarking tools. I would say this is close to maximum transfer rate for the ARM A78AE. Maybe an NVIDIA engineer knows with their internal testing but my links look to match the spec for PCIe gen4.

If you are having issues check if your pcie links are gen4.

Thanks @RDobitz for sharing this. Looks promising ;-)
Indeed, LnkCap shows that your device is detected as Gen4-able, and furthermore LnkSta just below shows that is indeed currently running in Gen4 mode.
Never seen it throttling back to Gen3 under heavy device usage ?
I have no experience with such Gen4 SSD device, but from what I’ve read it may draw and dissipate about 7W, and throttle back to lower if too hot, so my concern (I currently have no other platform for a Gen4 SSD in case it doesn’t work out).

@richg, this may provide an hypothesis for what you’re seeing. You may check with:

# install nvme-cli if not yet done:
sudo apt install nvme-cli

# Now you may check if supported with:
sudo nvme smart-log /dev/nvme0 | grep 'temperature'
1 Like

good thinking I didn’t think of checking that… That’s not slow, this drive was advertised as having 7000R/6000W and I’m getting 1700. It was advertised as being designed for PS5, Mac, & PC, but in general they were targeting the PS5 people. Do you know of anything else which is needed for enablement? I’ll do some digging, but afaik it was just in the linux kernel, but not sure. Worst case I’ll swap it out.

It was outfitted with a heatsink, and usually when these things overheat you would see it go faster then drop drastically, it seems like its working fine it just can’t break through that 1.7gb/s, I tried bumping the values up in the benchmark settings in disk it only made it more consistently stay at that level.

That’s weird, I tested it again and now it is faster… hmmm curious.

	Subsystem: Sony Corporation Device 9122
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 52
	Region 0: Memory at 2428000000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 16GT/s (ok), Width x4 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+
			 10BitTagComp+, 10BitTagReq-, OBFF Not Supported, ExtFmt+, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, TPHComp-, ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [d0] MSI-X: Enable+ Count=33 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [e0] MSI: Enable- Count=1/8 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [f8] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [110 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=10us PortTPowerOnTime=220us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [1e0 v1] Data Link Feature <?>
	Capabilities: [200 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [290 v1] Device Serial Number 00-00-00-00-00-00-00-00
	Capabilities: [2a0 v1] Power Budgeting <?>
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [340 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [378 v1] Lane Margining at the Receiver <?>
	Kernel driver in use: nvme

Still seems like it should be faster… I expect the 980 Pro is ideal but is there any other drives people have used and found to perform at over 4GB/s (as is that is too slow, well the faster it is the quicker you can load large models or use more performant swap.

Looks like throttling back…and forward.

Check what your device is detected to be able with:

 sudo lspci -vv | grep "NVMe" -A40 | grep -A1 LnkCap

should show 16 GT/s for Gen4

Check what speed it is currently running with:

sudo lspci -vv | grep "NVMe" -A40 | grep LnkSta:

Also check for errors if any with:

sudo lspci -vv | grep "NVMe" -A40 | grep AERCap