PCIe x4 Speed Issue with SSD

[BillyW]

Hi Everyone,

My setup is pretty simple: TX1 dev board + Samsung X941 AHCI SSD.
Drive is detected and working at x4, drive is tested on other systems with >1GB/s speeds.

When mounted to the TX1, the read speed I get is good, around 800MB/s, but the write speed is really bad, ~270MB/s.

Now, I saw some other folks have PCIe FPGA mounted with similar issue, see here:
https://devtalk.nvidia.com/default/topic/939528/jetson-tx1/pcie-data-transfer-speed/

So I wonder if this is a TX1 problem. Since the problem looks unidirectional.
I’m also getting an NVMe drive soon and I’ll post results if I see anything better.

Read Test:

ubuntu@tegra-ubuntu:/media/ubuntu/SamXP941AHCI5$ sudo hdparm -Tt /dev/sda
[sudo] password for ubuntu: 

/dev/sda:
 Timing cached reads:   3804 MB in  2.00 seconds = 1903.91 MB/sec
 Timing buffered disk reads: 1528 MB in  3.00 seconds = 509.11 MB/sec


ubuntu@tegra-ubuntu:/media/ubuntu/SamXP941AHCI5$ sudo /sbin/sysctl -w vm.drop_caches=3
vm.drop_caches = 3
ubuntu@tegra-ubuntu:/media/ubuntu/SamXP941AHCI5$ dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.22961 s, 873 MB/s

Write Test:

ubuntu@tegra-ubuntu:/media/ubuntu/SamXP941AHCI5$ sudo dd bs=1M count=256 if=/dev/zero of=test > ~/dd1M.txt
256+0 records in
256+0 records out
268435456 bytes (268 MB) copied, 0.998564 s, 269 MB/s

lspci -vv :

00:01.0 PCI bridge: NVIDIA Corporation Device 0fae (rev a1) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	Memory behind bridge: 13000000-130fffff
	Prefetchable memory behind bridge: 0000000020000000-00000000200fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: NVIDIA Corporation Device 0000
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [80] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd On, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v0] #00
	Capabilities: [140 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=30us PortTPowerOnTime=70us
	Kernel driver in use: pcieport

01:00.0 SATA controller: Samsung Electronics Co Ltd XP941 PCIe SSD (rev 01) (prog-if 01 [AHCI 1.0])
	Subsystem: Samsung Electronics Co Ltd Device a811
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 130
	Region 5: Memory at 13000000 (32-bit, non-prefetchable) 
	[virtual] Expansion ROM at 20000000 [disabled] 
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/2 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [d0] Vital Product Data
		Not readable
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [140 v1] Device Serial Number 00-00-00-00-00-00-00-00
	Capabilities: [150 v1] Power Budgeting <?>
	Capabilities: [160 v1] Latency Tolerance Reporting
		Max snoop latency: 256ns
		Max no snoop latency: 256ns
	Kernel driver in use: ahci

[kayccc]

Hi BillyW,

Thanks for raising this issue, and we’re doing the investigation now.
The status will be updated once we clarified it, please stay tuned.

Thanks

[BillyW]

Hi Kayccc,

Thanks for the response. Also, I like to update that I’ve compiled and installed the NVMe driver and did the same test (dd method), tested with Samsung XP951 NVMe version on nvme01 and still getting similar speeds, except read speed is now as slow as write (around 250MB/s). Its possible this is a PCIe hardware issue, can you let me know if this is the case?

Thanks,
Billy

[vidyas]

Hi,
We have seen write speeds of 475 MB/s with an OCZ SSD (AHCI based). So, it is unlikely this is a HW limitation.
Can you please give more info on the release version of L4T that you are using?

[BillyW]

Using L4T R23.2, do you have some details for the OCZ SSD testing?

[vidyas]

OCZ card details
01:00.0 Class 0100: Device 1b85:1041 (rev 02)

I’ve formatted the drive with ext2 file system and without any changes in L4T R23.2, I got around 250 MB/s (similar to your numbers)
But, after enabling EXT2 native support by enabling following configs in kerne, I could get around 480 MB/s
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y

if CONFIG_PREEMPT_TRACER=y is removed from configs, it went further up to touch 680 MB/s

[yahoo2016]

OCZ card details
01:00.0 Class 0100: Device 1b85:1041 (rev 02)

I’ve formatted the drive with ext2 file system and without any changes in L4T R23.2, I got around 250 MB/s (similar to your numbers)
But, after enabling EXT2 native support by enabling following configs in kerne, I could get around 480 MB/s
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y

if CONFIG_PREEMPT_TRACER=y is removed from configs, it went further up to touch 680 MB/s

Has the same drive been tested under L4T 24.1?

[vidyas]

Yes
With above config changes, I got around 680 MB/s write speed

[mpu]

Looks like I have a similar issue, but I’m still quite new to the Jetson and I don’t understand the fix. How do I enable the ‘configs in kerne’?

Any hints much appreciated.

[linuxdev]

@mpu:
Technically enabling those kernel configs is no different on Jetson than on other Linux systems (most kernel compile documentation applies even if not specifically for a Jetson…only kernel install really differs), but cross compile is required. Which L4T version are you using? On the Jetson you can find out with this:

head -n 1 /etc/nv_tegra_release

If R24.1 see this:
https://devtalk.nvidia.com/default/topic/936880/jetson-tx1/jetson-tx1-24-1-release-need-help-with-complier-directions-can-not-complie/post/4885136/#4885136

[mpu]

Thanks for pointing me to that post. Recompiling the kernel… excellent, seems painful. I’ll look into it.

[linuxdev]

Recompiling the kernel isn’t actually too bad. The part which is time consuming and requires details is putting the cross compilers on your host. Once the host has this it is trivial to make kernel changes.

[jholt]

For future reference, I had no luck with an OCZ RD400 512GB PCIe-based SSD with a Mailiya M.2 PCIe to PCIe 3.0 x4 Adapter in my Jetson TX1.

The device shows up with lspci:

# lspci
00:01.0 PCI bridge: NVIDIA Corporation Device 0fae (rev a1)
01:00.0 Non-Volatile memory controller: OCZ Technology Group, Inc. Device 6018 (rev 01)

But when I try to modprobe nvme, I get:

[  202.269298] nvme 0000:01:00.0: Device not ready; aborting initialisation

[vidyas]

Hi jholt,
Has this drive been tested on any other platform?
Can you try adding ‘pcie_aspm=off’ to kernel command line? (wondering if this is something to do with the ASPM states). Also please upload ‘lspci -s 01:00.0 -vvvv’ output.

[jholt]

Thanks vidyas! It works now. Here is lspci output in case that is useful to you.

Before adding pcie_aspm=off to /boot/extlinux/extlinux.conf APPEND line:

root@tegra-ubuntu:~# lspci -s 01:00.0 -vvvv
01:00.0 Non-Volatile memory controller: OCZ Technology Group, Inc. Device 6018 (rev 01) (prog-if 02 [NVM Express])
	Subsystem: OCZ Technology Group, Inc. Device 6018
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 130
	Region 0: Memory at 13000000 (64-bit, non-prefetchable) 
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s unlimited, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [178 v1] #19
	Capabilities: [198 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [1a0 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=400us
	Kernel driver in use: nvme
	Kernel modules: nvme

After adding that option and rebooting:

root@tegra-ubuntu:~# lspci -s 01:00.0 -vvvv
01:00.0 Non-Volatile memory controller: OCZ Technology Group, Inc. Device 6018 (rev 01) (prog-if 02 [NVM Express])
	Subsystem: OCZ Technology Group, Inc. Device 6018
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 130
	Region 0: Memory at 13000000 (64-bit, non-prefetchable) 
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s unlimited, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [178 v1] #19
	Capabilities: [198 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [1a0 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=400us
	Kernel driver in use: nvme
	Kernel modules: nvme

[jholt]

Here are iozone stats after mkfs -t ext4:

ubuntu@tegra-ubuntu:/mnt/ocz_ssd$ iozone -s 12g -r 16384 
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.429 $
		Compiled for 64 bit mode.
		Build: linux 

...
	Run began: Wed Feb  8 00:37:32 2017

	File size set to 12582912 kB
	Record Size 16384 kB
	Command line used: iozone -s 12g -r 16384
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
        12582912   16384   389367   742073   650548   585865   637334   701164   669400   1308254    632085   518285   626449   703909   717474

[jholt]

Oh, one other thing: nvme was not included with the stock kernel, so in order to make my OCZ rd400 work, I had to get the kernel source, copy over the /usr/src/linux-headers-3.10.96-tegra/.config to kernel/.config, make menuconfig, set ‘-tegra’ as the local version, enable the nvme module, make -j6 SUBDIRS=drivers/block modules, copy nvme.ko to /lib/modules/3.10.96-tegra/kernel/drivers/block/nvme.ko, depmod. Then I could modprobe nvme.

[Benjamin_Caro]

Hi,

Is there a consensus on what the best kernel config settings are and what the maximum write speed is?
Did anyone actually achieve 680MB/s?

I have a NVME drive that should be able to go to 1 GB/s but only see speeds of up to ~410MB/s.

I’m on a ext4 filesystem.

Thanks,

[vidyas]

In addition, you can try disabling ASPM, if not done already (adding ‘pcie_aspm=off’ to kernel command line would do that) and having ext2 file system.

[wugj]

OK I will try.