LI-AR0820-GMSL2 cameras stability issues

Hi everyone,

We bought 3 LI-AR0820-GMSL2 cameras and the necessary accessories to connect it to a Jetson AGX device, got the driver from Leopard Imaging, the cameras are working, but after some time, the Jetson AGX restarts and gets very slow. Even after reboot, it is still slow and it needs some time to get back to normal operation. Using Jetpack 4.4 because the driver supports that version. Have you experienced this kind of behavior before, or do you have any suggestions? Can it be overheating?

Tegrastats shows normal temperatures:
RAM 1632/31927MB (lfb 7459x4MB) SWAP 0/15964MB (cached 0MB) CPU [100%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] EMC_FREQ 0% GR3D_FREQ 0% AO@44.5C GPU@46C Tdiode@51C PMIC@100C AUX
@44.5C CPU@46C thermal@45.4C Tboard@46C GPU 0/592 CPU 466/586 SOC 2175/4052 CV 0/0 VDDRQ 310/666 SYS5V 1857/2132

I used jetson_multimedia_api/argus/samples/syncSensor for testingwith the modification of stream size to 4K.

Thank you,
Csaba

What’s if run argus_camera --module=3

Thanks for your reply @ShaneCCC!

I just tried it, but the symptoms are the same, even worse, after capturing a few image, it doesn’t have time to set the exposure correctly, application freezes and Jetson restarts. I will try to reinstall the whole system, if it helps or not.

UPDATE: after clean install the problem still persists, after some time argus_syncSensor freezes. “argus_camera --module=3” doesn’t work either.

@SimonZhu
Could you help to clarify this problem.

Thanks

@csabasipos89
Are you using this driver from below guide?
AR0820-GMSL2_R32.4.3_Xavier_NV-Quad_20200927_Driver_Guide.pdf

We didn’t experience the issue before. Is there any special operation (like install any software/libraries) before this issue occurred?
Please try below command to enable the fan manually to see if it helps.
sudo su
echo 255 > /sys/devices/pwm-fan/target_pwm

Yes @SimonZhu , we got the exact same document from Leopard Imaging, I strictly followed the guide, even downloaded the Jetson image from the link provided by the document. I did not install anything other than cuda-10.2 and tensorrt.

Here is what I did:

  1. Installed L4T and ar0820 driver as instructed by the document
  2. Installed CUDA from debs, downloaded by sdk manager for JetPack 4.4 (needed by the examples) (cuda-repo-l4t-10-2-local-10.2.89_1.0-1_arm64.deb)
  3. Installed cudnn and tensorrt from debs, downloaded by sdk manager for JetPack 4.4
  4. Downloaded jetson_multimedia_api_R32.4.3.tar.gz from dropbox, followed the link in the driver gide
  5. Compiled the argus example, with the modification of jetson_multimedia_api/argus/samples/syncSensor/main.cpp, I changed the stream resolution from 640x480 to 3840x2160: static const Size2D<uint32_t> STREAM_SIZE(640, 480); to static const Size2D<uint32_t> STREAM_SIZE(3840, 2160);

After that I have tried to run argus_camera --module=3 with 3 (ar0820) cameras attached, but it did not work (maybe a fourth camera needed, just a guess).

Then run the argus synSensor example: ./argus_syncsensor -t 3600

It starts and runs for a while, but after some time the Jetson AGX freezes and restarts and remains very slow, even ssh connection is laggy after that.

Thanks for your help.

@csabasipos89
Thanks for the information. I have a few questions.

  1. How long had all 3 cameras been running before the Jetson AGX freezed?
  2. Is the Xavier very hot when it freezes? If so, please check if the fan is working. If not, please use the command in my last comment to enable the fan to see if it helps.

@SimonZhu
It was varying how long they were running, the example runs 2 cameras only. Yes it was very hot, and I am not sure if the fans were working or not, but I did not hear them. I turned on the fan manually as you recommended and now it is running for almost an hour without any issue. I will test it a little more an get back to you. I will turn off the fans and test it again, if it freezes again or not.

@csabasipos89
Thanks. Look forward to the result.

@SimonZhu
So, I turned off the fan, after 20 minutes the Jetson crashed, and did not turned on the fan, with the fan on, it seems to be stable, after 3 hours running, no problem at all. Thanks for your help, it is good, it is stable now, but a little bit loudy :). Do you think, that the automatic fan control, depending on the temperature, can be fixed?
Thanks again.

@csabasipos89
Great to know you found the root cause. We will check the auto fan control function.

@SimonZhu
Thank you very much for your help!

@csabasipos89
We tested below driver again with 4 cameras running simultaneously, but the fan can work automatically after running the cameras a while.
All 4 cameras can run more than 4 hours without issues. Please double check if the auto fan function works on you side.

AR0820-GMSL2_R32.4.3_Xavier_NV-Quad_20200927_Driver_Guide.pdf

@SimonZhu
I haven’t tried 4 cameras just 2 or 3. I have double checked and the Jetson definitely fails after some time if I do not turn on the fan manually.

@csabasipos89
Can you check if the fan is turned on or not when you run 3 cameras for a while (before fails)?
What’s the ambient temperature in your testing? We did the testing in our office, and the room temperature is about 26℃.

@SimonZhu
OK, I will check again, but if I remember correctly when I tested, the fan didn’t turn on automatically. The room temperature it was tested in, is about 21-23 ℃.