R32.3.1.tx2-4g gstream app fails to run nvargus_daemon message

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

I am using a lumenera camera.

my gstreamer app will not start/restart 100% of the time, I found that restarting the nvargus_daemon after every run on my app helps but it is not 100%, using dmesg --follow will give me messages like:
[ 4116.848757] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 4116.848761] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 4116.848765] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 4116.848769] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 4116.848772] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 4116.848776] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 4116.848779] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[ 4116.848783] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[ 4116.848786] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[ 4116.848790] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[ 4116.848793] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[ 4116.848797] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[ 4116.848801] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[ 4116.848804] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[ 4116.848808] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[ 4116.848811] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[ 4116.848815] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[ 4116.848818] syncpt_thresh_cpu0_int_status(17) = 0x00000000

[ 4116.849069] ---- channels ----
[ 4116.849079]
channel 2 - 15820000.se

[ 4116.849082] NvHost basic channel registers:
[ 4116.849086] CMDFIFO_STAT_0: 00002040
[ 4116.849090] CMDFIFO_RDATA_0: 3340223b
[ 4116.849095] CMDP_OFFSET_0: 00000000
[ 4116.849099] CMDP_CLASS_0: 00000000
[ 4116.849102] CHANNELSTAT_0: 00000000
[ 4116.849105] The CDMA sync queue is empty.

[ 4116.849111]
channel 3 - 15830000.se

[ 4116.849113] NvHost basic channel registers:
[ 4116.849117] CMDFIFO_STAT_0: 00002040
[ 4116.849120] CMDFIFO_RDATA_0: 80232812
[ 4116.849125] CMDP_OFFSET_0: 00000000
[ 4116.849128] CMDP_CLASS_0: 00000000
[ 4116.849132] CHANNELSTAT_0: 00000000
[ 4116.849134] The CDMA sync queue is empty.

[ 4116.849140]
channel 4 - 15840000.se

Is there a better nvargus_daemon or some library available to make things more stable.

Terry

3

hello terrysu50z,

may I have more details about this lumenera camera, is this supported by Jetson Camera Partners?

there’re sync-point timeout errors from your kernel logs. it’s due to camera stack cannot receive frame signaling within a while.
it’s Sensor Pixel Clock for camera stack for calculation, you may review the sensor timing since you’re able to restart nvargus_daemon service to make it work sometimes.
thanks

4

Lumenera is a nvidia partner, it is a 8M camera.

I have seen isp in the dmesg’s might be releated to the 8M

Also reviewing forums, it might be related to cpu usage. My video app is using 180% of 4 cores, I have two turned off. So is the “nice” of nvargus_daemon maybe a factor here?

I forwarded the email to lumenera, hoping they will get back on the Sensor Pixel clock

Thanks,
Terry

5

hello terrysu50z,

could you please use top commands, and switch the mode to Irix mode by “I” (shift+i) to get average CPU usage.
please also refer to https://www.freebsd.org/cgi/man.cgi?top(1)
thanks

6

so my VideoSystem is using 44.3% using the “i” option, and nvargus-daemon is directly under it at 5.1%

Terry

7

hello terrysu50z,

before lumenera reply your question,
please check Applications Using V4L2 IOCTL Directly by using V4L2 IOCTL to verify basic camera functionality.
thanks

8

I ran the vctl-ctl command, when I did a ctrl-c I got
[ 1980.019899] ------------[ cut here ]------------
[ 1980.024534] WARNING: CPU: 0 PID: 12029 at drivers/media/v4l2-core/videobuf2-core.c:1667 __vb2_queue_cancel+0x11c/0x188
[ 1980.035367] —[ end trace 5d4aae212b22cc26 ]—

in the serial console, also ov5693.raw is zero length file.

what next?

Terry

9

hello terrysu50z,

v4l2-ctl is direct v4l2 interface to access the camera sensor, it’s used to validate sensor drivers.
so, here should be stability issue of lumenera camera. please contact with them for investigation.
thanks

11

after talking to the camera vendor, they pointed out that nvargus-daemon was crashing. Please fix nvargus-daemon in r32.3.1

12

hello terrysu50z,

you may upgrade the JetPack release to include the fixes.

FYI,
we’re having some fixes for camera stack, there’re changes included in the latest JetPack release.
please refer to r32.5 Release Notes for further details.
thanks

13

My camera driver only works and is released for on r32.3.1, so how do I update r32.3.1

Also other hardware support is only tested with r32.3.1, it is not possible to change releases at this time.

Thanks,
Terry

14

hello terrysu50z,

please check with sensor vendor, we had deliver error resilience patches to several camera partners based-on r32.3.1.
thanks

15

hello terrysu50z,

you may upgrade your JetPack release version to JetPack-4.5 / l4t-r32.5 to include error handling,
please refer to L4T Driver Package (BSP) Sources,
there’s added handling for the EVENT_TYPE_ERROR events, thanks

17

I can’t upgrade to r32.5, my camera driver is only good for r32.3.1

can you tell me and the camera vendor more about enhancement from r32.5 that need to be back ported to r32.3.1

Do these changes go in the camera driver?

Do these changes go in the OS?

Please give more details about what to look at in the BSP sources,

Can you forward me the info that you sent to the camera partners? I might have to do the changes the vendor gave me the camera driver code, so I think I might need to make the changes.

I did not write the camera driver, I have no information that a camera partner would have, I am experienced in linux drivers and such, but I am a camera newbee.

Thanks,
Terry

18

hello terrysu50z,

please download this package, Argus_Error_Resiliency_phase2.7z (2.5 MB)

there’re kernel patches, you may integrate the changes and rebuilt/update the kernel;
please check the binaries under /usr/lib/, you should replace those binaries and perform a warm-reboot to make it works.
please note that, it’s only works for JetPack-4.3 / l4t-r32.3.1, please apply those for verification.
thanks

19

@JerryChang ,

I very carefully put in the kernel changes and backed up the lib files, and when I rebooted, my MIPI DSI display was black and I am really in a bind.

What in those libs could of killed my MIPI DSI display.

Terry

20

hello terrysu50z,

I don’t think those patches effect your display driver, as you can see, those were updates to the VI, and camera driver.
please share your kernel messages for more details, thanks

21

@JerryChang
added some debug to dsi side, rebuilt kernel, installed it, rebooted and Mipi DSI display came up, appears the first reboot after adding patches and libs, failed to read the device tree nvidia,dsi-reset element. It now does and more reboots are booting.

The patches might or might not have helped. I still am not 100%. I still see the messages from above when trying to start my app. It does seem to be less often, but it still does happen. Timing problem?

Next suggestion please.

Where these patches really available back in 5/2020, if so why am I just hearing about the patches now?

Thanks,
Terry

22

You better sharing some dmesg instead of only saying something not work. We don’t know what kind of error you have now. Share the log and save your own time.

23

hello terrysu50z,

as I mentioned before, it’s patches based-on r32.3.1; JetPack release is now available for JetPack-4.5.1 / l4t-r32.5.1
please share your kernel messages for more details, thanks