Watchdog for detecting a faulty camera

[severin2]

Hi
I’m composing a total of 6 GMSL cameras into one encoded stream for low latency internet streaming. We need to implement safety functions such as detecting when one camera gets disconnected (bad cable, connector, internal error etc.) and therefore displaying a message on a custom GUI when one or more cameras are not sending data. We found the watchdog plugin, and have tried to implement it in our pipe directly after each nvv4l2camerasrc but with no outputting error message when a camera is unplugged.
Does it require an end of stream for the watchdog to kick in? When trying only a simple pipe it does not work either:

gst-launch-1.0 -m nvv4l2camerasrc device=/dev/video0 ! watchdog timeout=5000 ! "video/x-raw(memory:NVMM), format=UYVY, width=1920, height=1080" ! nvvidconv ! "video/x-raw(memory:NVMM), format=RGBA" ! nvdrmvideosink sync=false

[linuxdev]

I can’t give you a specific answer, but the way watchdog works is that there is a file in “/dev” (some “/dev/watchdog*” file). Once watchdog starts, the file requires something be written to it (usually anything, e.g., the number 1, so long as it writes to it like an ordinary file) over some time period (e.g., 30 seconds). So when the write waits too long to occur, the system assumes failure and reboots.

Multiple methods of writing are available, sometimes from kernel space (a driver might do this), but usually from user space (the “watchdog”). So what you have to do is:

• Enable watchdog.
• Write to watchdog within the allotted periodic time.
• Detect the presence of the working camera as the method of writing, detecting this faster than the watchdog rate.
• Use that detection to perform that periodic write.
• Avoid write if not detected as functioning.
• Normally you would need this write watchdog to function immediately on startup, or your system is going to fail for not getting the write fast enough.
• Be ready to manually echo something (like “sudo echo 1 > /dev/watchdog*”) if you must manually prevent shutdown. Your watchdog program could have an alternate “service mode” to detect a file or something used to say to “just always run” if being worked on.

[DaneLLL]

Hi,
We don’t have experience about using wathdog plugin. Would see if other users can share experience. One suggestion is that you may try v4l2src plugin and see if the error case can be detected. Please try like:

v4l2src ! watchdog ! ...

[severin2]

Hi

Link to watchdog element: watchdog

We’ve been doing some more testing and research. It seems like the watchdog don’t like the nvcompositor element, with an overlaying sink (nvdrmvideosink).

With fakesink and dump=true, we still see incoming data even with camera unplugged after stream has been running a little time. Its basically looking exactly the same as in attached picture, except freeing pipeline is not appearing, and pipeline still runs with a small amount of sporadic data coming.

Test 1:

• We tested running only a single pipe with one camera going to fakesink. Watchdog works just as supposed to, it triggers an error message (displayed with -m) which is what we want.
• By setting the watchdog timeout too low, the stream may not start. We kind of found a sweetspot with 500ms timeout.

gst-launch-1.0 -m nvv4l2camerasrc device=/dev/video0 ! watchdog timeout=500 ! "video/x-raw(memory:NVMM), format=(string)UYVY" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)RGBA" ! fakesink dump=true

Test 2:

• We also tested a similar stream to our final solution (more cameras) with nvcompositor and two cameras. By sending the composed stream to a fakesink, the watchdog works.
• With two cameras and nvcompositor, the watchdog seems to detect no write in pipeline startup, and the pipeline may have to be restarted a few times to finally start playing.

gst-launch-1.0 -m nvcompositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 sink_0::zorder=1 \
sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540 sink_1::zorder=2 \
! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)RGBA, height=(int)1080, width=(int)1920" ! fakesink dump=true \
nvv4l2camerasrc device=/dev/video0 ! watchdog timeout=500 ! "video/x-raw(memory:NVMM), format=(string)UYVY" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)RGBA" ! comp. \
nvv4l2camerasrc device=/dev/video1 ! watchdog timeout=500 ! "video/x-raw(memory:NVMM), format=(string)UYVY" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)RGBA" ! comp.

Test 3:

• The watchdog also works with one single camera ending up in nvdrmvideosink.
• Also here the watchdog will not detect any writing unless we restart the stream a few times. 700ms worked out, however sometimes the stream starts playing just fine on the first try.

gst-launch-1.0 -m nvv4l2camerasrc device=/dev/video0 ! watchdog timeout=700 ! "video/x-raw(memory:NVMM), format=(string)UYVY" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)RGBA" ! nvdrmvideosink

Test 4:

• The result we want to achieve is that the watchdog detects if one camera fails, by adding a watchdog to each camerasource.
• The watchdog can not affect our pipeline because of the need of high performance (realtime streaming) and therefore we’ll rather not use watchdog over using non Nvidia plugins in our pipeline.
• The following pipeline is one example of what we want to use.
• If one camera gets unplugged when running the following pipeline, nothing happens. The stream stops, however data will sporadically come from the unplugged camera, e.g. making the still video jump/glitch in intervals of approximately one second.

gst-launch-1.0 -m nvcompositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 sink_0::zorder=1 \
sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540 sink_1::zorder=2 \
! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)RGBA, height=(int)1080, width=(int)1920" ! nvdrmvideosink sync=false \
nvv4l2camerasrc device=/dev/video0 ! watchdog timeout=500 ! "video/x-raw(memory:NVMM), format=(string)UYVY" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)RGBA" ! comp. \
nvv4l2camerasrc device=/dev/video1 ! watchdog timeout=500 ! "video/x-raw(memory:NVMM), format=(string)UYVY" ! nvvidconv ! "video/x-raw(memory:NVMM), format=(string)RGBA" ! comp.

Hope this clear things up, perhaps we’ve even been doing something completely wrong thats also an easy mistake. We just want to hear if anybody else face the same problems with watchdog, if not we’ll simply not use it and eventually wait until watchdog + compositor + videosink support has been added (if its not supported). Adding a screenshot from terminal when watchdog has been triggered.