Slow ssd write speed (px2, Ubuntu, AutoChauffeur)

Autonomous Vehicles DRIVE PX 2 DRIVE Hardware
1

We are faced with the problem that the transfer speeds of a hard disk connected via USB3.0 do not achieve sufficient results on the PX2.
Hardware used:

NVIDIA DRIVE 5.0 Linux SDK: https://docs.nvidia.com/drive/nvvib_docs/index.html#page/NVIDIA%2520DRIVE%2520Linux%2520SDK%2520Development%2520Guide%2FAppendix%2Fcomms_usb_performance.html%23

https://devtalk.nvidia.com/default/topic/1007100/faq/data-logging/

The maximum transfer rate achieved was 240 MB/s for copying the driveworkssdk to the SSD, and 130MB/s for measuring with “dd”, with a minimum transfer rate of 400-500MB/s in each of these sources.
Several formatting formats were also tested and did not lead to any improvement.

On a USB3.0-enabled laptop with Ubtunu 16.04 LTS with the latest drivers/upgrades, however, the desired 500MB/s could be consistently achieved.

Please let us know if there are updates for new USB drivers, where they will be provided and how to install them. An apt-get update && upgrade was not executed at any time and should not be necessary.

Thank you and best regards

2

Dear randtie,

Did you test USB bandwidth via any USB hub?
If yes, for testing could you connect SSD directly to tegra A with no hub? And connect hub to tegra b with keyboard and mouse, SSH to tegra a for read/write testing.

$ dd if=/dev/zero of=tmp.tmp bs=1G count=10

3

Dear Steve,

We tested your proposal on four PX2 units, but there was no improvement.

4

Dear randtie,

What about the file system on the storage drive?
If NTFS, please use ext4 with noatime and try it. Thanks.

5

We already tested that. No significant improvement.

6

Dear randtie,

Could you please refer to below link for your topic? Thanks.

/usr/local/driveworks-1.2/doc/nvdwx_html/dwx_faq.html on DPX2
How to achieve the maximum USB 3.0 write throughput for data acquisition?

/usr/local/driveworks-1.2/doc/nvdwx_html/dwx_recording_devguide_high_throughput_recording.html on DPX2

7

Dear SteveNV,

After the update we were able to increase the data rate to up to 306Mb/s.
Setup still as described above, what other changes would be possible for an improvement? Would a more detailed description than in the documentation be possible?

Thank you

8

Dear PversusNP,

Firstly, please update DriveWorks SDK to the latest version(1.2.400).
And then please refer to High Throughput Recording part in DriveWorks doc.
DriveWorks SDK Reference → Tutorials → Recording Sensor Data → High Throughput Recording

9

Dear SteveNV,

As described, the update has enabled us to achieve a speed gain of 50 percent and we have strictly adhered to the documentation. (DriveWorks SDK Reference → Tutorials → Recording Sensor Data → High Throughput Recording).
Are even more adjustments possible to reach the 400-500MB/s?
Thank you.

10

Still having the same problems, also working on Driveworks 1.2.400 and working based on the documentation.

11

Dear PversusNP & randtie,

Do you use USB hub when testing SSD bandwidth?
If yes, please try to connect SSD directly to TegraA with no hub and connect hub to TegraB with keyboard and mouse, SSH to TegraA a for read/write testing. Thanks.

12

Hello SteveNV,
We are using SSH exactly like you described it.
Greetings

13

Hi,

we have just reproduced your issue. I am raising this bug internally now and will come back to you as soon as I have an update for.

In general ~300MB/s is the expected throughput, so I will check if that’s a documentation bug.

  • Fabian
14

Hi Randtie,
hi PversusNP (nice nick btw),

could you please provide me your .json files you have used for recording?

Currently, I am doing even more triage and benchmarks on it.

  • Fabian
15

Hey Fabian,

we are currently using no .json file for recording and checking the writing speed just
with the dd-command.
Do you think the logging process is faster than the dd-command?

Tank you.

16

Hi PversusNP,

the dd-command should reach up to 500Mb/s and I do not see any performance gain in using logging.

We are still investigating in this issue and I will come back to you as soon as I have more information.

  • Fabian
17

Hi guys,

We have done some benchmarks and the maximum speed you can get now is around 300MB/s.

Unfortunately the DRIVE documentation is wrong and this will be corrected.

Fabian

18

Hi,
Thanks for the testing.
Is this issue fixed on the Pegasus-Board?
Kind regards,
randtie

19

Hi @randtie, since I struggle with some PX2 data transfer issues as well, did you get an answer to your question about Pegasus?