I’m working with a Jetson running L4T R32.4.4 (JetPack 4.4.1) on a Jetson Nano. I’m using 4K@30FPS and 2x HD @ 10FPS cameras via Argus/GStreamer pipelines.
After long streaming sessions (sometimes 10 minutes, sometimes longer), the pipelines crash with the following nvargus report:
NVMAP_IOC_WRITE failed: Interrupted system call
(payload:1): GStreamer-CRITICAL **: 20:31:59.750: gst_mini_object_unlock:
assertion ‘(state & access_mode) == access_mode’ failed
nvbuf_utils: dmabuf_fd 1055 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
NvBufferGetParams failed for src_dmabuf_fds[2]
nvbuffer_composite Failed
In dmesg there is no additional information, camera driver just report that it stops streaming (because of nvargus crash). after restarting nvargus and pipeline stream works fine (until next crash occurs).
I didn’t observe anything strange,i start tracing after starting the pipeline and first message i saw was immediately after crash and it was telling that it shutting down the camera, so sa me as in dmesg.
I monitored DMA buffer usage in /sys/kernel/debug/dma_buf/bufinfo: the count typically sits around 600–750, with a maximum around 830 before crashes. The is not always above 800 when crash occurs.
Every consumer in my GStreamer pipelines starts with a queue configured as queue max-size-buffers=4 leaky=downstream flush-on-eos=TRUE.
So far, I cannot find a way to trace which buffer is being held or released too late, and I suspect it may be a buffer lifetime / leak issue in Argus on this older release.
Questions:
Is this a known Argus/DMA-BUF issue in L4T R32.4.4 / JetPack 4.4.1?
Are there recommended workarounds besides upgrading (e.g., GStreamer pipeline tweaks, Argus config changes, consumer queue settings, CMA tuning, etc.)?
How can I get more detailed debug information about Argus buffer management on this release?
Any pointers, release notes, or forum links would be very helpful.
The pipeline looks bit less stable (I have more crushes).
And in the nvargus still outputs when stream crashes.
Wrong frequency range!
Wrong frequency range!
NVMAP_IOC_WRITE failed: Interrupted system call
(payload:1): GStreamer-CRITICAL **: 08:14:23.862: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
nvbuf_utils: dmabuf_fd 1064 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
NvBufferGetParams failed for src_dmabuf_fds[0]
nvbuffer_composite Failed
We run pipeline from cpp sources. Its relatively complex pipeline with several camera sources and several sinks (udp, file, ros) which are dynamically changing the pipeline based on user settings.
So sorry I can’t provide you simple gst-lunch command.
This patch should be applied on gstnvarguscamerasrc.cpp. In our pipeline we have following components:
-GstNvArgusCameraSrc
-GstCapsFilter
-GstTee
-GstAppSrc
-Gstnvvconv
-GstFakeSink
-GstQueue
-GstVideoRate
-Rosimagesink
-GstNVCompositor
-GstOMXH264Enc-omxh264enc
-MpegTsMux
-GstUDPSink
Could it be that the other component of the pipeline is causing the problem with buffer? Are you aware about similar issue in other components (eg GstNVCompositor)?
The issue is present when we add second HD camera.
If we run 4K@30FPS and one HD@10FPS it’s okay. But when we add second HD@10FPS , after some time we get Can not get HW buffer from FD… But it’s not immediate it’s after start of the stream, it takes several minutes. Also The error happens with all three cameras when we are changing the pipeline in runtime.
Hey Shane,
thanks for the suggestion, it would be bit more complex for us to modify the driver and dts + whole pipeline to downgrade 4K to FullHD.
That’s why instead we tried to modified the pipeline. Surprisingly after removing GstNVCompositor from the pipeline. We did not observe crash of the stream Can not get HW buffer from FD... Exiting....
This is the part of our pipeline which we removed.
