DeepStream v5 bug | NvDufSurface memory leakage

This problem occurs in DeepStream 5 only, and not occur in DeepStream 4

In the reference application deepstream_test5_app, i tried to use the function callback overlay_graphics() to draw custom overlays. In that function I map the gestreamer input buffer in this sequence:

  1. gst_buffer_map()
  2. NvBufSurfaceMap()
  3. NvBufSurfaceSyncForCpu()

Then I draw some overlays with standard openCV function calls and release the buffer as follows:

  1. NvBufSurfaceSyncForDevice()
  2. NvBufSurfaceUnMap()
  3. gst_buffer_unmap

Calling NvBufSurfaceMap() and NvBufSurfaceUnMap() in that callback causes a memory leak of about 100 kilobytes per frame. The application then crashes after some time with the following output:
PosixMemMap:71 [12] mmap failed
nvbufsurface: NvBufSurfaceMap function failed…
nvbufsurface: mapping of buffer (0) failed
nvbufsurface: error in mapping
PosixMemMap:71 [12] mmap failed
PosixMemMap:71 [12] mmap failed
PosixMemMap:71 [12] mmap failed
App run failed

The same procedure works perfectly if I draw the overlays in the callback function bbox_generated_probe_after_analytics().I would prefer to use overlay_graphics(), because there I get direct access to an RGBA buffer instead of a NV12 buffer in bbox_generated_probe_after_analytics.

1 Like

Hey Customer,
Could you fill the setup info firstly?

• Hardware Platform (Jetson / GPU) - Xavier NX
• DeepStream Version - 5.0
• JetPack Version (valid for Jetson only) - 4.4 DP
• TensorRT Version - 7.0
• NVIDIA GPU Driver Version (valid for GPU only)

Thanks, we should support both RGBA and NV12, have you checked dsexample -> get_converted_mat?
And would you mind to share your code with us if the reference code no help.

I too am having the same problem. However, my deepstream version is 4.0.1. Can you tell me what exactly should I look for in dsexample? I am using it to save the detections and it works properly on local videos and even rtsp streams to an extent. And randomly PosixMemMap:71 [12] mmap failed comes up.
Interestingly, I am still able to see the osd bounding boxes and the tiled display working properly.

Hi sharanssundar,

Please open a new topic for your issue. Thanks

While deepstream-test5 is running, the memory leak can be observed by using the pmap command. After some time (some minutes, up to one hour, depending on free memory) the application fails with the messages ‘Warning - Can’t Map NvBufSurface.’ and 'Warning - Can’t sync NvBufSurface for CPU

the patch file attached,
thanks

reproduce_memleak_in_overlay_graphics_cb_patch_in_deepstream-test5.txt (2.5 KB)

Hey,

I have exactly the same problem. I was also able to reproduce the memory leak in test app 5 with the patch posted by by dvir. Is there any solution from NVIDIA?

Thanks

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Can you repro your issue with pure ds-test5?

I am having exactly the same problem. Are there any workarounds?
Thanks

Hi infol7p51,

Please help to open a new topic for your issue. Thanks

Hi,

I am facing the same memory leakage problem. I wrote a c++ plugin to rotate a frame by a particular angle. For that purpose I used:

NvBufSurfaceMap(&ip_surf, 0, 0, NVBUF_MAP_READ_WRITE)

to map the buffer and

NvBufSurfaceUnMap(&ip_surf, 0, 0)

to release memory again where

NvBufSurface ip_surf;
ip_surf = *surface;
ip_surf.numFilled = ip_surf.batchSize = 1;
ip_surf.surfaceList = &(surface->surfaceList[i]).

If I remove theses lines memory does not increase on CPU.
So NvBufSurfaceUnMap does not release the memory.

Could you please provide an approach to solve this problem.

Thank you in advance for your help,

**• Hardware Platform Jetson TX2 **
• DeepStream Version 5.0
• JetPack Version 4.4

Hi zerodeterminant

Please help to open a new topic for your issue. Thanks

Hi,

I finally solved this problem, i.e. I found a workaround. By adapting the code of gst-dsexample I created a new NvBufSurface object when I load my custom plugin (in that way we have an internal buffer). In the next step I use NvBufSurfaceCopy in order to copy a new buffer to the internal buffer .

Then apply NvBufSurfaceMap(config->internal_buffer, 0, 0, NVBUF_MAP_READ_WRITE) and NvBufSurfaceSyncForCpu(config->internal_buffer, 0, 0) such that the buffer can be accessed by the CPU (use WRITE if you want to write the Buffer back in memory).

After that step you can manipulate the buffer and with NvBufSurfaceSyncForDevice(config->internal_buffer, 0, 0) you write it back to the GPU and by using NvBufSurfaceCopy(config->internal_buffer, &ip_surf) != 0) you copy the internal buffer to the orginal buffer.

In addition to that, I would like to mention that there is still a constant memory leakage of 100kB if we use NvBufSurfaceMap and you still cannot release the memory by calling NvBufSurfaceUnmap (However, the memory only increases once by 100kB and that’s it because we always overwrite the internal buffer).

1 Like

Hi, zerodeterminant.
Can you show me the source code you implemented?

Hi all,
I was facing the same memory leak problem. Basically, on my laptop with an RTX graphic card everything runs fine (at least for an hour), but same code in Jetson NX was crashing after 15-20 min consistently.
I had two problems:

  1. I wasn’t doing the Unmapping, so, in my “gst_mymodule_transform_ip” function, I added the unmaping call right after NvSurfaceSyncForDevice call:
    NvBufSurfaceSyncForDevice (surface, -1, -1);
    if (NvBufSurfaceUnMap (surface, -1, -1)){
    goto error;
    }

  2. My second mistake was regarding the indices. I was using zero indices because I thought it was the default, NvBufSurfaceSyncForDevice (surface, 0, 0); But it turns out that would only affect first batch, first plane. Indices must be -1 to deal with all batches and all planes.

After those changes, it seems to work fine, after 2 hours of processing, it still runs without problems.

For the NVIDIA guys: it would be nice to have a more comprehensive documentation about memory handling in deepstream modules. The dsexample helps, but is not helpful to get started, and it doesn’t tell much about the logic unless you studied it for a few hours.

@zerodeterminant could you please share the code of your workaround? I´m facing the same issue described here and i´m trying to do your solution, but after doing the copy of the surface with NvBufSurfaceCopy((NvBufSurface *) in_map_info.data, inter_buf), the NvBufSurfaceMap is not working and crashing the app.

@kayccc or @bcao is Nvidia working on this problem? The leak exists also in Deepstream 5.0.1 with latest Jetpack.