Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU): Jetson Xavier NX • DeepStream Version: 6.0 • JetPack Version (valid for Jetson only) 4.6.1 • TensorRT Version: 8.0 (CUDA 10.2) • Issue Type( questions, new requirements, bugs): Bug • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing): buffer_error.py (2.0 KB)
Starting gstreamer pipeline using the following string: videotestsrc num-buffers=-1 ! video/x-raw ! nvvideoconvert ! nvstreammux0.sink_0 nvstreammux width=640 height=480 batch-size=1 ! buffer-error ! fakesink sync=false
Where buffer-errror is the custom python element that extracts the image (see attachement) into a numpy array similar to the example app: Image data access application except extracting it each step and it being an element.
After ~64500 images the following error pops up, however it does not hard crash the pipeline:
PosixMemMap:71 [12] mmap failed
nvbufsurface: NvBufSurfaceMap function failed
nvbufsurface: mapping of buffer (0) failed
nvbufsurface: error in mapping
get_nvds_buf_Surface: Failed to map buffer to CPU
Looking at the memory usage using free -m only ~2GB is used and ~6GB is free.
I could not reproduce the error on a normal GPU.
The error seems to happen at the line that extracts the frame_pointer:
The string I provided was the minimal string with which I could reproduce the error. In our actual pipeline it happens after a nvstreamdemux element. Looking at nvstreamdemux (src pad) with gst-inspect they give the exact same output as far as I can tell.
If I understand correctly, the sample app extracts the frames to python after the nvmultistreamtiler, which also has as output “video/x-raw(memory:NVMM)”.
I am also confused why it does work for the first ~65k frames on the jetson and on a “normal” GPU it does not crash (for at least 100k frames). Does this mean that it is possible to do it this way on a normal GPU?
edit:
I further investigated the difference between jetson and normal gpu and I found something intresting.
On jetson the virtual memory keeps increasing upto ~ 90g after which it stops and prints the errors.
On gpu the virtual memory does not increase and stays at ~8g, which seems a lot more reasonable.
As I mentioned the pipeline works on a normal gpu, so our plugin works under those limitation. If you have looked at the python code I provided which should reproduce the error, it does not even need to use the output for the virtual memory creep to start.
I tried to replicate the error on the sample app, however that was not successful, but it did help met to narrow it down to these pipelines: