NVBufUtils DMABUF_FD not found

[youngy]

Hey all.

I have a system working using (mostly) the v4l2 interface to pass frames into and to withdraw them from the hardware encoder. This is using memory mapped buffers for both capture and output planes, following the tegra_multimedia_api example carefully. It operates fine during standard operation, however, when I have finished encoding a stream and attempt to de-allocate the buffers, I (sometimes) run into the following error.

nvbuf_utils: dmabuf_fd 1248 mapped entry NOT found.

Before I call v4l2_close, I dequeue each of the v4l2_buffers and have un-mapped the memory that they are using. Not that I believe it is anything to do with the capture or output buffers, because the fd stated is not one that is being allocated during my memory mapping. The sequence I take before closing the device is as follows.

• Turn off output stream using VIDIOC_STREAMOFF.
• Dequeue all output buffers using VIDIOC_DQBUF.
  This doesn’t actually do anything because all the buffers seem to be automatically dequeued by VIDIOC_STREAMOFF
• Unmap all of the memory using munmap with the appropriate memory pointers.
• Deallocate all output buffers using VIDIOC_REQBUFS and a request count of 0.
  This is where the encoder seems to be breaking.
• Repeat the process for capture stream.
• Close the device using v4l2_close()

I’ve read the following link and the linked solution within it and I do not believe it applies in my case (it is hard to tell because the test file in the original topic is unable to be downloaded).

nvbuf_utils: dmabuf_fd 1286 mapped entry NOT found

Hi all, I am developing a gstreamer application on Jetson TX2 which is supposed to process video coming from a CSI camera. At one point in my pipeline, I get a segmentation fault and the program crashes with the following message. According to gdb the crash occurs somewhere here: The strange thing is that the calling code looks like this: void fillWorkingBuffer(GstBuffer* gst_buffer, int working_fd) { GstMapInfo in_map_info; if (gst_buffer_map(gst_buffer, &in_map_info, GST_MAP_REA…

Does anyone have any suggestions as to what could be happening to cause this?

Edit: Clarify close sequence.
Edit: Correct close sequence.

[youngy]

After some further debugging and some further looking into the multimedia API, I now send a frame with 0 bytes used into the encoder and receive the 0 bytes encoded frame before performing the shutdown procedure. I’ve also found the function that is causing the seg fault using gdb.

nvbuf_utils: dmabuf_fd 1172 mapped entry NOT found
Thread 5 “compress” received signal SIGSEGV, Segmentation fault.
[Switching to LWP 3286]
0x0000007f98c05c2c in NvBufferMemUnMap () from /usr/lib/libnvbuf_utils.so.1.0.0
(gdb) up
#1 0x0000007f94186388 in ?? () from /usr/lib/libtegrav4l2.so
(gdb) up
#2 0x0000007f9418b110 in ?? () from /usr/lib/libtegrav4l2.so
(gdb) up
#3 0x0000007f941807c4 in ?? () from /usr/lib/libtegrav4l2.so
(gdb) up
#4 0x0000007f94174960 in TegraV4L2_Ioctl () from /usr/lib/libtegrav4l2.so
(gdb) up
#5 0x0000007f941b0e78 in plugin_ioctl ()
from /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so
(gdb) up
#6 0x0000007f98c1d5c0 in v4l2_ioctl () from /usr/lib/libv4l2.so.0

[youngy]

Upon further investigation, I’ve found what was causing the library to try and free a non-existent dma buffer, although I don’t really understand why. In my dequeue thread, I was using select() to query when the encoder was ready for me to dequeue the buffer.

After swapping over to V4L2_CID_MPEG_VIDEO_DEVICE_POLL, this no longer occurred, however the time to encode a frame increased from 6ms to 40ms. As this was unacceptable, I’ve ended with a compromise of using select() for the runtime operation, but when I close the encoder, I dequeue all capture buffers using the V4L2_CID_MPEG_VIDEO_DEVICE_POLL method.

However, extended testing of this problem has led to other issues.

Over the course of opening → encoding → closing the encoder ~ 300 times, the memory usage of the system increases until it reaches about 3.5G used (2.5 of this seems to be the encoder), then it crashes and refuses to encode any more frames, throwing a whole lot of warning messages.

Opening in O_NONBLOCKING MODE
Opening in O_NONBLOCKING MODE
NvMMLiteOpen : Block : BlockType = 8
NvMMLiteOpen : Block : BlockType = 8
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8
NvMMLiteBlockCreate : Block : BlockType = 8
NVMEDIA: NvMMAttributeVideoEnc_QPRange Get Parameter
VENC: NvMMLiteVideoEncSetAttribute: 4543:  NvMMLiteAttributeVideoEnc_QPRange: Bad Parameter
NVMEDIA: NvMMAttributeVideoEnc_QPRange Get Parameter
VENC: NvMMLiteVideoEncSetAttribute: 4543:  NvMMLiteAttributeVideoEnc_QPRange: Bad Parameter
NVMEDIA: H265 : Profile : 1
NVMEDIA: H265 : Profile : 1
[  839.132964] iommu_context_dev 13e10000.host1x:ctx2: pin_array_ids: could not map attachment err=-12
[  839.132991] iommu_context_dev 13e10000.host1x:ctx2: pin_array_ids: could not map attachment err=-12
[  839.132997] falcon 15340000.vic: nvhost_ioctl_channel_submit: failed with err -12
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
[  839.133049] falcon 15340000.vic: submit_add_gathers: failed to copy user inputs: class_ids=0000000080a3d038 num_cmdbufs=2
[  839.133051] falcon 15340000.vic: nvhost_ioctl_channel_submit: failed with err -22
[  839.142579] iommu_context_dev 13e10000.host1x:ctx2: pin_array_ids: could not map attachment err=-12
[  839.142586] falcon 154c0000.nvenc: nvhost_ioctl_channel_submit: failed with err -12
[  839.142597] falcon 154c0000.nvenc: submit_add_gathers: failed to copy user inputs: class_ids=0000000080a29038 num_cmdbufs=2
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 13, SyncPointValue = 0)
[  839.142599] falcon 154c0000.nvenc: nvhost_ioctl_channel_submit: failed with err -22
[  839.225692] falcon 15340000.vic: nvhost_ioctl_channel_submit: failed with err -12
[  839.235305] falcon 15340000.vic: submit_add_gathers: failed to copy user inputs: class_ids=0000000080a33038 num_cmdbufs=2
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
[  839.248938] falcon 15340000.vic: nvhost_ioctl_channel_submit: failed with err -22
[  839.259753] iommu_context_dev 13e10000.host1x:ctx2: pin_array_ids: could not map attachment err=-12
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 13, SyncPointValue = 0)
[  839.271986] falcon 154c0000.nvenc: nvhost_ioctl_channel_submit: failed with err -12
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
[  839.287550] falcon 154c0000.nvenc: submit_add_gathers: failed to copy user inputs: class_ids=0000000080a15038 num_cmdbufs=2
[  839.301811] falcon 154c0000.nvenc: nvhost_ioctl_channel_submit: failed with err -22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 11, Sync PointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 12, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 13, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 13, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 12, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 12, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 12, SyncPointValue = 0)

It repeats this until the encoder is closed and I attempt to reopen it, before following the whole process again.

[youngy]

Switching to using V4L2_MEM_DMABUF on the output plane resolved this as per

NvVideoEncoder repeated start-stop causes crash and memory leak

Hi DaneLLL, Hmmm, strange. I will reflash 28.1 on my Jetson and give looped 01_video_encode a try with the linked library later. We are close to production, and we already sent out some test systems. Anyway, we decided to migrate to 28.2.1 because stability is critical for our customers and I am almost sure that we can fix the memory leak issue on 28.2.1. Beyond that, we have one more major issue which could be on the Tegra side.