Slow JetsonNano When Writing Files

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

Hello,

I’m working on a project with a NVIDIA JetsonNano 4GB developer board on which an old Jetpack 3.1 with CUDA 10 runs. In the project I’m capturing images with the FLIR Spinnaker SDK at 13 fps. These images get processed in the following threads

  • Thread 1 capturing
  • Thread 2 undistortion
  • Thread 3 Tiny YOLO V2 with TensorRT from this example
  • Thread 4 Sending detected BBoxes from YOLO using UART
  • Thread 5 JPEG encoding
  • Thread 6 JPEG storing

Basically this pipeline works pretty well and the workload of the Jetson Nano is pretty low. But sometimes the pipeline gets stuck which can take between 100ms to 5 seconds in worst case. I can see this in a Log which I created for debugging. During this time it seems like my program is doing absoulutely nothing for no reason. The problem also only occurs if the JPEG storing is enabled. The JPEG encoding itself does’t seem to have a very bad influence on the duration of my pipeline. We currently store every second image. If no image is stored the pipeline takes 43-44 ms, if an image is stored (and encoded) it takes around 50-53 ms. Our current goal is to stay below 100ms better 80ms. It would also be ok if the image storing and encoding takes longer and some of the images are not stored.

To fix the problem we already tried many different appraches including:

  • Testing different mass storage (USB-Stick, External SSD (Samsung T7, Sand Disk Extreme PRO), MicroSD).
  • Split up writing the image in small parts with breaks of 2 ms.
  • Uninstalling the Userinterface
  • Disabling different services (Ubuntu whoopsie, apt autoupgrade etc.)
  • Disabling SWAP / ZRAM - We need less than 2GB of RAM but have 4GB
  • Test ext4 and exfat filesystem for USB3.0 SSD and pendrives

Also, I was trying to examine the problem with Linux perf tools, but I wasn’t able to install them since of problems/missing Kernelheaders.

The USB-SSDs and pendrives are mounted with the sync option, which should force immediate file writing onto the drive.

Basically what I need is a OS with realtime operating system capabilities. But with such a low workload on the JetsonNanoa I would expect that it is possible to achieve always less than 100ms for a pipeline pass through.

Any ideas what could go wrong here?

Thanks a lot in advance for your help and time.

Stephan

3

Hi,
Please provide your release version($ head -1 /etc/nv_tegra_release). We don’t support Jetson Nano in JetPack 3.1. Would like to confirm the release version first.

4

Thank you for your fast reply. The output of the command is:

# R32 (release), REVISION: 3.1, GCID: 18186506, BOARD: t210ref, EABI: aarch64, DATE: Tue Dec 10 06:58:34 UTC 2019

This could be a problem. But so far it worked well. I will have a look at the different versions.

5

Hi,
Your release is Jetpack4.3. There is a known issue about JPEG encoding. Please apply the prebuilt lib in
Jetson/L4T/r32.3.x patches - eLinux.org
[GSTREAMER]streaming using jpegenc halts after a short delay

And check if it helps your usecase.

For running deep learning inference, we have DeepStream SDK. There is demonstration of Yolo models and you may take a look and give it a try.

6

Thanks. I will probably be able to test the patch tomorrow with my teammate. I will let you know if it worked.

In a feature release we will have a closer look at Deepstream. Thanks.