I’m working on a project with a NVIDIA JetsonNano 4GB developer board on which an old Jetpack 3.1 with CUDA 10 runs. In the project I’m capturing images with the FLIR Spinnaker SDK at 13 fps. These images get processed in the following threads
- Thread 1 capturing
- Thread 2 undistortion
- Thread 3 Tiny YOLO V2 with TensorRT from this example
- Thread 4 Sending detected BBoxes from YOLO using UART
- Thread 5 JPEG encoding
- Thread 6 JPEG storing
Basically this pipeline works pretty well and the workload of the Jetson Nano is pretty low. But sometimes the pipeline gets stuck which can take between 100ms to 5 seconds in worst case. I can see this in a Log which I created for debugging. During this time it seems like my program is doing absoulutely nothing for no reason. The problem also only occurs if the JPEG storing is enabled. The JPEG encoding itself does’t seem to have a very bad influence on the duration of my pipeline. We currently store every second image. If no image is stored the pipeline takes 43-44 ms, if an image is stored (and encoded) it takes around 50-53 ms. Our current goal is to stay below 100ms better 80ms. It would also be ok if the image storing and encoding takes longer and some of the images are not stored.
To fix the problem we already tried many different appraches including:
- Testing different mass storage (USB-Stick, External SSD (Samsung T7, Sand Disk Extreme PRO), MicroSD).
- Split up writing the image in small parts with breaks of 2 ms.
- Uninstalling the Userinterface
- Disabling different services (Ubuntu whoopsie, apt autoupgrade etc.)
- Disabling SWAP / ZRAM - We need less than 2GB of RAM but have 4GB
- Test ext4 and exfat filesystem for USB3.0 SSD and pendrives
Also, I was trying to examine the problem with Linux perf tools, but I wasn’t able to install them since of problems/missing Kernelheaders.
The USB-SSDs and pendrives are mounted with the sync option, which should force immediate file writing onto the drive.
Basically what I need is a OS with realtime operating system capabilities. But with such a low workload on the JetsonNanoa I would expect that it is possible to achieve always less than 100ms for a pipeline pass through.
Any ideas what could go wrong here?
Thanks a lot in advance for your help and time.