Imbalanced Performance between Read and Write Performance

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

I installed a Samsung 970 EVO NVMe M.2 SSD to the M.2 port on Jetson Xavier by following the instruction on Installing an NVMe SSD on Nvidia Jetson Xavier | Medium. I created ext4 file system on the SSD drive.

nvidia@jetson-0423018054929:/xavier_ssd$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 28G 17G 9.5G 64% /
devtmpfs 7.7G 0 7.7G 0% /dev
tmpfs 7.8G 26M 7.8G 1% /dev/shm
tmpfs 7.8G 36M 7.7G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/nvme0n1p1 458G 73M 434G 1% /xavier_ssd

I got an unbalanced read and write performance and the performance is a factor of 4~6 slower than the speed of Samsung 970 EVO (3500MB/second Read and 2500MB/second write). i need the speed to simulate high speed data coming to Jetson for inference.

Any tuning is needed on Jetson Xavier?

The following is the write performance.

nvidia@jetson-0423018054929:/xavier_ssd$ time dd if=/dev/zero of=./100G count=10485760 bs=10240
10485760+0 records in
10485760+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 338.842 s, 317 MB/s

real 5m38.856s
user 0m18.692s
sys 5m8.980s

The following is the read performance:

nvidia@jetson-0423018054929:/xavier_ssd$ time dd of=/dev/null if=./100G count=10485760 bs=10240
10485760+0 records in
10485760+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 141.971 s, 756 MB/s

real 2m21.983s
user 0m18.336s
sys 2m1.672s

2

dtyu,

Which power mode are you using? Have you enabled jetson_clock.sh?

3

Dear WayneWWW:

After I turn on the ./jetson_clocks.sh and set mode 0. I doubled the performance. However, it is still far away from the spec. Did I hit the limit of Xavier? Is the M2 port on PCI Gen2 bus or Gen3 bus? What is the physical limitation of the M2 port? If I mount the SSD to the PCI Gen3X16 bus on Xavier? Will I improve the performance?

Thanks
Dantong

nvidia@jetson-0423018054929:~$ sudo ./jetson_clocks.sh --show
SOC family:tegra194 Machine:jetson-xavier
Online CPUs: 0-7
CPU Cluster Switching: Disabled
cpu0: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1190400
cpu1: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1497600
cpu2: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600
cpu3: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600
cpu4: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1190400
cpu5: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600
cpu6: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600
cpu7: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1728000
GPU MinFreq=114750000 MaxFreq=1377000000 CurrentFreq=114750000
EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=1331200000 FreqOverride=1
Fan: speed=255

nvidia@jetson-0423018054929:/xavier_ssd$ time dd if=/dev/zero of=./100GByte count=1048576 bs=102400
1048576+0 records in
1048576+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 176.06 s, 610 MB/s

nvidia@jetson-0423018054929:/xavier_ssd$ time dd of=/dev/null if=./100GByte count=1048576 bs=102400
1048576+0 records in
1048576+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 73.439 s, 1.5 GB/s

4

If this is using the PCIe of the m.2 (and isn’t USB), then the device should show up in:

lspci

Just as an example, if the device shows up with slot “01:00.0”, then you could create a verbose listing of PCIe details for just that device via:

sudo lspci -s 01:00.0 -vvv

You might want to post the output of that. A verbose listing would show device capabilities alongside actual settings (sometimes a device is capable of more than the bus is currently set to).

5

Dear LinuxDev,
Thanks for replying. Here is the output.

nvidia@jetson-0423018054929:~$ lspci
0000:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0000:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a808
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 (rev 13)
nvidia@jetson-0423018054929:~$ sudo lspci -s 01:00.0 -vvv
[sudo] password for nvidia:
0000:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a808 (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd Device a801
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 32
Region 0: Memory at 38200000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s unlimited, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
Vector table: BAR=0 offset=00003000
PBA: BAR=0 offset=00002000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158 v1] Power Budgeting <?>
Capabilities: [168 v1] #19
Capabilities: [188 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [190 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=40us
Kernel driver in use: nvme

0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 (rev 13) (prog-if 01 [AHCI 1.0])
Subsystem: Marvell Technology Group Ltd. Device 9171
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 588
Region 0: I/O ports at 100010
Region 1: I/O ports at 100020
Region 2: I/O ports at 100018
Region 3: I/O ports at 100024
Region 4: I/O ports at 100000
Region 5: Memory at 30210000 (32-bit, non-prefetchable)
Expansion ROM at 30200000 [disabled]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: 91201000 Data: 0000
Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Kernel driver in use: ahci
Kernel modules: ahci

nvidia@jetson-0423018054929:~$

6

I am checking the test result on our side. Thanks for your patience.

7

Just FYI, the NVMe controller is PCIe rev. 3., and it is running at full speed. So PCIe is in theory capable of running fast enough and has not throttled back to an earlier revision.

Probably not related: The Marvell SATA controller is PCIe rev. 2, but has throttled back to rev. 1. If the data goes through the Marvell SATA controller, then this would be a data throughput bottleneck.

8

Thanks for reply.
Do you have suggestions on how to tune this SSD?

Thanks Dantong

9

We should wait to see what @WayneWWW finds out. Keep in mind there is a holiday here this week, so it might slow down research. All I can say is that the NVMe PCIe communications is not throttling back to slower speeds and thus PCIe is not likely to be any kind of bottleneck.

10

Tried with dd command and the write performance indeed very slow.
Could you share the benchmark of your card on other platform?

11

Could you also try other I/O tools to test?

12

Sometimes I wonder if dd itself might be slow. When you use dd to write direct to a device special file you bypass any slowdown associated with the ext4 drivers. If you want to bypass both ext4 drivers and any unknown dd inefficiency, then you could do something like this for an alternate read test after you use the nvpmodel -m 0 and jetson_clocks.sh (this example assumes “/dev/nvme0n1p1”):

sudo time cat /dev/nvme0n1p1 > /dev/null

If you don’t mind wiping the entire disk you could cat “/dev/zero” to “/dev/nvme0n1p1”. I say “entire disk” because I’m not sure if this would stop with the named partition or not. I also don’t know how an NVMe reserves space, and although on a regular disk this would not touch earlier boot cylinders perhaps this would harm content outside of the one parition…don’t know.

13

The read performance does not improve comparing with dd.

Here is the result. I do not have any other computer that has M2 port.

sudo time cat /dev/nvme0n1p1 > /dev/null

nvidia@jetson-0423018054929:~$ sudo su -
root@jetson-0423018054929:~# time cat /dev/nvme0n1p1 > /dev/null

real 4m36.364s
user 0m3.752s
sys 3m59.800s

root@jetson-0423018054929:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 28G 17G 9.4G 64% /
devtmpfs 7.7G 0 7.7G 0% /dev
tmpfs 7.8G 4.0K 7.8G 1% /dev/shm
tmpfs 7.8G 36M 7.7G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/nvme0n1p1 458G 73M 434G 1% /xavier_ssd
tmpfs 1.6G 132K 1.6G 1% /run/user/1001
/dev/mmcblk1p1 59G 53M 56G 1% /media/nvidia/datadisk
tmpfs 1.6G 0 1.6G 0% /run/user/0
root@jetson-0423018054929:~# bc -l
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty’.
458/(4*60+36)
1.65942028985507246376

14

I also noticed that the same writing problem with the internal SSD on Xavier. Therefore, the problem might be with the software or configuration or file system (EXT4) configuration.

nvidia@jetson-0423018054929:~/temp$ pwd
/home/nvidia/temp

nvidia@jetson-0423018054929:~$ cd temp/
nvidia@jetson-0423018054929:~/temp$ time dd if=/dev/zero of=./10Gpb bs=1048576 count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 61.848 s, 139 MB/s

real 1m1.854s
user 0m0.016s
sys 0m7.540s

nvidia@jetson-0423018054929:~/temp$ time dd if=/dev/zero of=./10Gpb bs=1048 count=8192000
8192000+0 records in
8192000+0 records out
8585216000 bytes (8.6 GB, 8.0 GiB) copied, 60.3912 s, 142 MB/s

real 1m0.395s
user 0m5.000s
sys 0m29.048s

15

I also wonder dd is the right tool for doing this. The performance varies on my AGX with different input size when using dd.

Could you try to download iozone3 and give it a try?

16

Hi,

Any update for this issue?

17

Tested Samsung 970EVO 1TB.

hdparm
write: 1.6 GB/s

nvidia@xavier:/SSD$ sudo hdparm -t --direct /dev/nvme0n1

/dev/nvme0n1:
 Timing O_DIRECT disk reads: 4960 MB in  3.00 seconds = 1653.07 MB/sec

iozone
Max write: 2.7 GB/s

nvidia@xavier:/SSD$ iozone -aR -i 0 -I -q 10m -g 1g -f /SSD/iozone

"Writer report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"
"64"   26227  57446  95230  195680  179819
"128"   44227  62017  108026  224181  274641  549056
"256"   45117  63667  180527  215705  531343  646518  879535
"512"   61406  104086  182720  213344  355790  557092  909640  1292259
"1024"   54392  110320  204475  263304  455950  817963  1166221  1120288  1652033
"2048"   65682  134004  229543  387880  669477  891644  1167617  1280015  1491617  1647852
"4096"   49288  98110  178155  343282  503073  835929  1136249  1405098  1673830  2013626  2314159
"8192"   69858  135319  230883  404981  617572  1051727  1194008  1456118  1686631  2020233  2346046  2588342
"16384"   71868  137823  247637  388920  667835  946669  1283497  1603766  1663877  1949278  2548865  2697440
"32768"   74115  145641  254007  346051  692095  940475  1313988  1549746  1712916  2157468  2459325  2677566
"65536"   73039  138831  243799  410289  548693  827998  1327740  1618768  1767899  2206068  2495944  2420087
"131072"   73969  149054  260274  443047  715673  977550  1153711  1600137  1810683  2169707  2521829  2675109
"262144"   77953  149159  294531  477643  595763  1071843  1345190  1621636  1767837  2167841  2460589  2621306
"524288"   78769  147300  278412  510483  833789  1182904  1561934  1855780  1860490  2388000  2598894  2611971
"1048576"   74971  155531  292412  479970  809291  1179634  1546891  1830614  2034060  2404312  2600383  2608553

Environment

JetPack 4.1.1 DP

root@xavier:/home/nvidia# sudo ./jetson_clocks.sh --show
SOC family:tegra194  Machine:jetson-xavier
Online CPUs: 0-7
CPU Cluster Switching: Disabled
cpu0: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1728000
cpu1: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1804800
cpu2: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1344000
cpu3: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1190400
cpu4: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600
cpu5: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1420800
cpu6: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1420800
cpu7: Gonvernor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1344000
GPU MinFreq=318750000 MaxFreq=1377000000 CurrentFreq=318750000
EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=800000000 FreqOverride=0
Fan: speed=0

Unfortunately, I am having errors. On the boot screen I see message:

[FAILED] Failed to start Load Kernel Modules.

root@xavier:/home/nvidia# systemctl status systemd-modules-load.service
● systemd-modules-load.service - Load Kernel Modules
   Loaded: loaded (/lib/systemd/system/systemd-modules-load.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2018-12-13 03:11:33 UTC; 2h 9min ago
     Docs: man:systemd-modules-load.service(8)
           man:modules-load.d(5)
  Process: 2376 ExecStart=/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
 Main PID: 2376 (code=exited, status=1/FAILURE)

Though not directly related with the SSD, these errors started after SSD installation.
Reflashing Xavier with a new Jetpack 4.1.1. didn’t help.

18

The speed of write seems better using iozone. Is there any problem on the I/O performance?

Please file a new topic for the kernel error.

19

I haven’t used SSD that much yet, and no problems so far. Though I do worry about overheating.

New topic is here: [FAILED] on boot screen - Jetson AGX Xavier - NVIDIA Developer Forums
Thank you!

20

I do not know how to interpret the performance. The maximum initial write 800MB/s. and maximum rewrite is 1.2GB/second with 100thread.

nvidia@jetson-0423018054929:/media/nvidia/datadisk$ sudo iozone -i 0 -t 1
Iozone: Performance Test of File I/O
Version $Revision: 3.429 $
Compiled for 64 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
             Vangel Bojaxhi, Ben England, Vikentsi Lapa.

Run began: Fri Dec 14 08:21:51 2018

Command line used: iozone -i 0 -t 1
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 1 process
Each process writes a 512 kByte file in 4 kByte records

Children see throughput for  1 initial writers 	=  356026.66 kB/sec
Parent sees throughput for  1 initial writers 	=   26948.71 kB/sec
Min throughput per process 			=  356026.66 kB/sec 
Max throughput per process 			=  356026.66 kB/sec
Avg throughput per process 			=  356026.66 kB/sec
Min xfer 					=     512.00 kB

Children see throughput for  1 rewriters 	=  141005.33 kB/sec
Parent sees throughput for  1 rewriters 	=   20738.85 kB/sec
Min throughput per process 			=  141005.33 kB/sec 
Max throughput per process 			=  141005.33 kB/sec
Avg throughput per process 			=  141005.33 kB/sec
Min xfer 					=     512.00 kB

nvidia@jetson-0423018054929:/media/nvidia/datadisk$ sudo iozone -i 0 -t 2
Iozone: Performance Test of File I/O
Version $Revision: 3.429 $
Compiled for 64 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
             Vangel Bojaxhi, Ben England, Vikentsi Lapa.

Run began: Fri Dec 14 08:21:30 2018

Command line used: iozone -i 0 -t 2
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 2 processes
Each process writes a 512 kByte file in 4 kByte records

Children see throughput for  2 initial writers 	=  454543.23 kB/sec
Parent sees throughput for  2 initial writers 	=   35554.01 kB/sec
Min throughput per process 			=  196581.22 kB/sec 
Max throughput per process 			=  257962.02 kB/sec
Avg throughput per process 			=  227271.62 kB/sec
Min xfer 					=     392.00 kB

Children see throughput for  2 rewriters 	=  474763.25 kB/sec
Parent sees throughput for  2 rewriters 	=   39835.11 kB/sec
Min throughput per process 			=  219923.67 kB/sec 
Max throughput per process 			=  254839.58 kB/sec
Avg throughput per process 			=  237381.62 kB/sec
Min xfer 					=     444.00 kB

nvidia@jetson-0423018054929:/media/nvidia/datadisk$ sudo iozone -i 0 -t 10
Iozone: Performance Test of File I/O
Version $Revision: 3.429 $
Compiled for 64 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
             Vangel Bojaxhi, Ben England, Vikentsi Lapa.

Run began: Fri Dec 14 08:22:26 2018

Command line used: iozone -i 0 -t 10
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 10 processes
Each process writes a 512 kByte file in 4 kByte records

Children see throughput for 10 initial writers 	=  722789.12 kB/sec
Parent sees throughput for 10 initial writers 	=   17474.83 kB/sec
Min throughput per process 			=   49618.28 kB/sec 
Max throughput per process 			=  110656.21 kB/sec
Avg throughput per process 			=   72278.91 kB/sec
Min xfer 					=     312.00 kB

Children see throughput for 10 rewriters 	=  676453.32 kB/sec
Parent sees throughput for 10 rewriters 	=   32584.58 kB/sec
Min throughput per process 			=   27741.09 kB/sec 
Max throughput per process 			=  105960.63 kB/sec
Avg throughput per process 			=   67645.33 kB/sec
Min xfer 					=     148.00 kB

nvidia@jetson-0423018054929:/media/nvidia/datadisk$ sudo iozone -i 0 -t 20
Iozone: Performance Test of File I/O
Version $Revision: 3.429 $
Compiled for 64 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
             Vangel Bojaxhi, Ben England, Vikentsi Lapa.

Run began: Fri Dec 14 08:22:39 2018

Command line used: iozone -i 0 -t 20
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 20 processes
Each process writes a 512 kByte file in 4 kByte records

Children see throughput for 20 initial writers 	=  888997.52 kB/sec
Parent sees throughput for 20 initial writers 	=   17721.92 kB/sec
Min throughput per process 			=       0.00 kB/sec 
Max throughput per process 			=  170618.64 kB/sec
Avg throughput per process 			=   44449.88 kB/sec
Min xfer 					=       0.00 kB

Children see throughput for 20 rewriters 	= 1286784.50 kB/sec
Parent sees throughput for 20 rewriters 	=   31863.75 kB/sec
Min throughput per process 			=       0.00 kB/sec 
Max throughput per process 			=  169402.36 kB/sec
Avg throughput per process 			=   64339.23 kB/sec
Min xfer 					=       0.00 kB

iozone test complete.
nvidia@jetson-0423018054929:/media/nvidia/datadisk$ sudo iozone -i 0 -t 100
Iozone: Performance Test of File I/O
Version $Revision: 3.429 $
Compiled for 64 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
             Vangel Bojaxhi, Ben England, Vikentsi Lapa.

Run began: Fri Dec 14 08:24:28 2018

Command line used: iozone -i 0 -t 100
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 100 processes
Each process writes a 512 kByte file in 4 kByte records

Children see throughput for 100 initial writers 	=  853107.40 kB/sec
Parent sees throughput for 100 initial writers 	=    3425.13 kB/sec
Min throughput per process 			=       0.00 kB/sec 
Max throughput per process 			=  175454.91 kB/sec
Avg throughput per process 			=    8531.07 kB/sec
Min xfer 					=       0.00 kB

Children see throughput for 100 rewriters 	=  883088.80 kB/sec
Parent sees throughput for 100 rewriters 	=   58175.61 kB/sec
Min throughput per process 			=       0.00 kB/sec 
Max throughput per process 			=  123193.41 kB/sec
Avg throughput per process 			=    8830.89 kB/sec
Min xfer 					=       0.00 kB