This problem occurs in DeepStream 5 only, and not occur in DeepStream 4
In the reference application deepstream_test5_app, i tried to use the function callback overlay_graphics() to draw custom overlays. In that function I map the gestreamer input buffer in this sequence:
gst_buffer_map()
NvBufSurfaceMap()
NvBufSurfaceSyncForCpu()
Then I draw some overlays with standard openCV function calls and release the buffer as follows:
NvBufSurfaceSyncForDevice()
NvBufSurfaceUnMap()
gst_buffer_unmap
Calling NvBufSurfaceMap() and NvBufSurfaceUnMap() in that callback causes a memory leak of about 100 kilobytes per frame. The application then crashes after some time with the following output:
PosixMemMap:71 [12] mmap failed
nvbufsurface: NvBufSurfaceMap function failedā¦
nvbufsurface: mapping of buffer (0) failed
nvbufsurface: error in mapping
PosixMemMap:71 [12] mmap failed
PosixMemMap:71 [12] mmap failed
PosixMemMap:71 [12] mmap failed
App run failed
The same procedure works perfectly if I draw the overlays in the callback function bbox_generated_probe_after_analytics().I would prefer to use overlay_graphics(), because there I get direct access to an RGBA buffer instead of a NV12 buffer in bbox_generated_probe_after_analytics.
Thanks, we should support both RGBA and NV12, have you checked dsexample ā get_converted_mat?
And would you mind to share your code with us if the reference code no help.
I too am having the same problem. However, my deepstream version is 4.0.1. Can you tell me what exactly should I look for in dsexample? I am using it to save the detections and it works properly on local videos and even rtsp streams to an extent. And randomly PosixMemMap:71 [12] mmap failed comes up.
Interestingly, I am still able to see the osd bounding boxes and the tiled display working properly.
While deepstream-test5 is running, the memory leak can be observed by using the pmap command. After some time (some minutes, up to one hour, depending on free memory) the application fails with the messages āWarning - Canāt Map NvBufSurface.ā and 'Warning - Canāt sync NvBufSurface for CPU
I have exactly the same problem. I was also able to reproduce the memory leak in test app 5 with the patch posted by by dvir. Is there any solution from NVIDIA?
There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks
I finally solved this problem, i.e. I found a workaround. By adapting the code of gst-dsexample I created a new NvBufSurface object when I load my custom plugin (in that way we have an internal buffer). In the next step I use NvBufSurfaceCopy in order to copy a new buffer to the internal buffer .
Then apply NvBufSurfaceMap(config->internal_buffer, 0, 0, NVBUF_MAP_READ_WRITE) and NvBufSurfaceSyncForCpu(config->internal_buffer, 0, 0) such that the buffer can be accessed by the CPU (use WRITE if you want to write the Buffer back in memory).
After that step you can manipulate the buffer and with NvBufSurfaceSyncForDevice(config->internal_buffer, 0, 0) you write it back to the GPU and by using NvBufSurfaceCopy(config->internal_buffer, &ip_surf) != 0) you copy the internal buffer to the orginal buffer.
In addition to that, I would like to mention that there is still a constant memory leakage of 100kB if we use NvBufSurfaceMap and you still cannot release the memory by calling NvBufSurfaceUnmap (However, the memory only increases once by 100kB and thatās it because we always overwrite the internal buffer).
Hi all,
I was facing the same memory leak problem. Basically, on my laptop with an RTX graphic card everything runs fine (at least for an hour), but same code in Jetson NX was crashing after 15-20 min consistently.
I had two problems:
I wasnāt doing the Unmapping, so, in my āgst_mymodule_transform_ipā function, I added the unmaping call right after NvSurfaceSyncForDevice call:
NvBufSurfaceSyncForDevice (surface, -1, -1);
if (NvBufSurfaceUnMap (surface, -1, -1)){
goto error;
}
My second mistake was regarding the indices. I was using zero indices because I thought it was the default, NvBufSurfaceSyncForDevice (surface, 0, 0); But it turns out that would only affect first batch, first plane. Indices must be -1 to deal with all batches and all planes.
After those changes, it seems to work fine, after 2 hours of processing, it still runs without problems.
For the NVIDIA guys: it would be nice to have a more comprehensive documentation about memory handling in deepstream modules. The dsexample helps, but is not helpful to get started, and it doesnāt tell much about the logic unless you studied it for a few hours.
@zerodeterminant could you please share the code of your workaround? IĀ“m facing the same issue described here and iĀ“m trying to do your solution, but after doing the copy of the surface with NvBufSurfaceCopy((NvBufSurface *) in_map_info.data, inter_buf), the NvBufSurfaceMap is not working and crashing the app.
@kayccc or @bcao is Nvidia working on this problem? The leak exists also in Deepstream 5.0.1 with latest Jetpack.