GStreamer PosixMemMap cannot allocate memory

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hello - I’ve seen a few messages in these forums about getting PosixMemMap:84 mmap failed : Cannot allocate memory from gstreamer pipelines, and haven’t really found any answers. I’ve managed to get this error after running the following command for about 1 hour:

gst-launch-1.0 \  videotestsrc is_live=1 pattern=ball  ! video/x-raw, width=3840, height=2160, format=UYVY, framerate=10/1 ! nvvidconv flip-method=2 ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\), format=NV12, width=320, height=180 ! nvjpegenc ! filesink location=/dev/null \
  videotestsrc is_live=1 pattern=ball  ! video/x-raw, width=3840, height=2160, format=UYVY, framerate=10/1 ! nvvidconv flip-method=2 ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\), format=NV12, width=320, height=180 ! nvjpegenc ! filesink location=/dev/null \
  videotestsrc is_live=1 pattern=ball  ! video/x-raw, width=3840, height=2160, format=UYVY, framerate=10/1 ! nvvidconv flip-method=2 ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\), format=NV12, width=320, height=180 ! nvjpegenc ! filesink location=/dev/null \
  videotestsrc is_live=1 pattern=ball  ! video/x-raw, width=3840, height=2160, format=UYVY, framerate=10/1 ! nvvidconv flip-method=2 ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\), format=NV12, width=320, height=180 ! nvjpegenc ! filesink location=/dev/null \
  videotestsrc is_live=1 pattern=ball  ! video/x-raw, width=3840, height=2160, format=UYVY, framerate=10/1 ! nvvidconv flip-method=2 ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\), format=NV12, width=320, height=180 ! nvjpegenc ! filesink location=/dev/null \
  videotestsrc is_live=1 pattern=ball  ! video/x-raw, width=3840, height=2160, format=UYVY, framerate=10/1 ! nvvidconv flip-method=2 ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\), format=NV12, width=320, height=180 ! nvjpegenc ! filesink location=/dev/null \
  videotestsrc is_live=1 pattern=ball  ! video/x-raw, width=3840, height=2160, format=UYVY, framerate=10/1 ! nvvidconv flip-method=2 ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\), format=NV12, width=320, height=180 ! nvjpegenc ! filesink location=/dev/null \
  videotestsrc is_live=1 pattern=ball  ! video/x-raw, width=3840, height=2160, format=UYVY, framerate=10/1 ! nvvidconv flip-method=2 ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\), format=NV12, width=320, height=180 ! nvjpegenc ! filesink location=/dev/null

In other words, 8 copies of an identical stream that does
videotestsrc ! nvvidconv ! nvvidconv ! nvjpegenc ! filesink

The nvvidconv elements just move memory to NVMM, rotate, and change the image format. I’m obviously not using any cameras here. It would be very helpful to understand why the stream is attempting to allocate any memory at all once it starts - I would expect each element to allocate memory up front, and then just reuse buffers with each new image that moves through the pipeline.

If you wonder why I’m doing this with 8 streams - the intent here is to use 8 cameras, and a more complicated pipeline for each, but I’m trying to narrow down the problem as much as I can.

any help would be appreciated, thanks!

3

Hi,
There are NvBufsurface allocated in nvvidconv, so it looks like memory is insufficient in the use-case. If you use Jetpack 6.1 or 6.2, please download the packge to get source code of nvvidconv plugin:

Jetson Linux | NVIDIA Developer
Driver Package (BSP) Sources

and check how it runs. You can follow README to manually build it.

4

I’m on an earlier jetpack, but will see what I can do. Meanwhile: how much memory is necessary? The device I’m using has 64 GB RAM and the process is not using close to that. Also, it runs for about an hour before emitting these messages, which suggests something slowly leaking.

5

Hi,
It may be leak on the Jetpack version. You may try later version. For information, which version are you using?

6

We’re running jetpack 5.1.1.

7

Update on this - the gst-launch-1.0 process is calling:
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_SHARED, 259, 0) = 0xfffc931c0000
multiple times per second, without ever calling munmap. This is happening in each thread that is actually running one of the sub-pipelines (I had to strace the PID of the thread, rather than the primary PID of the process, since the main thread is just sleeping) . The PosixMemMap failure occurs once the number of outstanding mmaps (visible via /proc//maps) hits the kernel limit of vm.max_map_count which on this machine is currently 65530:

$ sysctl vm.max_map_count
vm.max_map_count = 65530

So clearly the problem is a leak, not a lack of sufficient memory.

8

all right, finally resolved this - it appears to be a problem in nvjpegenc, which doesn’t let go of mmapped buffers. apparently this was fixed in jetpack 5.1.2:

thanks for the help, DaneLLL

1 Like
10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.