Nvargus-daemon stop by event overflow at about 64975 frames

[fumiya.fujinaka]

Hi,

I testing camera pipeline with nvarguscamerasrc.
As a result of experiments, in the case of a single camera, the next image cannot be obtained at the about 67975th frame.
The frame count that stops is the same if I change the frame rate or image size.
However, I suspect that this element is not the direct cause, but that there is a problem on the nvargus-daemon side.
Could you please investigate this error?

OS: L4T35.3.1
SoM: Orin NX 16GB
powermode: MAX N
GStreamer pipeline: nvarguscamerasrc → nvjepgenc → videorate → queue → tcpserversink(or fakesink)

Error message from nvarguscamerasrc:

PosixMemMap:84 mmap failed : Cannot allocate memory
(Argus) Error InsufficientMemory:  (propagating from src/eglstream/ImageImpl.cpp, function initialize(), line 455)
(Argus) Error InsufficientMemory:  (propagating from src/eglstream/ImageImpl.cpp, function copyToNvBuffer(), line 496)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadExecute:653 IImageNativeBuffer not supported by Image.
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:247 (propagating)

nvargus-daemon log:

Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: === camera-evt-server[4884]: CameraProvider destroyed (0xffff7c6c2850)=== camera-evt-server[4884]: Connection closed (FFFF8366E900)=== camera-evt-server[4884]: Connection cleaned up (FFFF8366E900)=== camera-evt-server[5255]: Connection established (FFFF8366E900)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: NvPclHwGetModuleList: No module data found
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: NvPclHwGetModuleList: No module data found
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
Jun 28 06:57:20 orin-nx20 nvargus-daemon[946]: === camera-evt-server[5255]: CameraProvider initialized (0xffff7cb76500)CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
Jun 28 07:24:35 orin-nx20 nvargus-daemon[946]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
Jun 28 07:24:35 orin-nx20 nvargus-daemon[946]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
Jun 28 07:24:35 orin-nx20 nvargus-daemon[946]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
Jun 28 07:24:35 orin-nx20 nvargus-daemon[946]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
Jun 28 07:24:35 orin-nx20 nvargus-daemon[946]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
Jun 28 07:24:35 orin-nx20 nvargus-daemon[946]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)
Jun 28 07:24:35 orin-nx20 nvargus-daemon[946]: (Argus) Error OverFlow: Too many pending events, ignoring new events (in src/api/EventProviderImpl.cpp, function addEvent(), line 158)

[ShaneCCC]

Maybe remove the nvjepgenc to clarify.

nvarguscamerasrc → videorate → queue → fakesink

[fumiya.fujinaka]

Hi, @ShaneCCC

Thanks for your advice.

I tried that and any pipeline:

1. nvarguscamerasrc → fakesink
2. nvarguscamerasrc → nvvidconv → fakesink
3. nvarguscamerasrc → nvv4l2h264enc → fakesink
4. nvarguscamerasrc → videorate → queue → fakesink

All pipeline exceeded 64975.
Does nvjpegenc have a limit on the number of frames?

[DaneLLL]

Hi,
Thaks for reporting it. This should be same as
Issues with jpegenc

We are investigating it and shall have it fixed in next release.

[fumiya.fujinaka]

Hi, @DaneLLL

Thanks for your information.
I understand, I’m looking forward to release of the fixed version!

[Honey_Patouceul]

This might not be optimal, but as workaround you may try using system memory rather than NVMM memory as nvjpegenc input:
nvarguscamerasrc → nvvidconv → video/x-raw,format=I420 → nvjpegenc → …
Seems the 16 bits overflow issue only bites in NVMM input branch.

[fumiya.fujinaka]

I see, there is a cause in the branch of NVMM.
It worked beyond 64975 Frames by changing to SystemMemory!
It’s not quite the framerate I expected, but it’s good enough as a workaround.

Thanks for your advice!

[Honey_Patouceul]

Also faced similar error when running (with v4l2loopback driver version 0.12.3 built with dkms) :

gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1280,height=720,format=I420' ! nvjpegenc ! multipartmux ! multipartdemux single-stream=1 ! 'image/jpeg, parsed=(boolean)true, width=(int)1280, height=(int)720, colorimetry=(string)2:4:7:1, framerate=(fraction)30/1,sof-marker=(int)0' ! v4l2sink device=/dev/video0 -v
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)640, height=(int)480, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)640, height=(int)480, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)3/4, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)3/4, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstNvJpegEnc:nvjpegenc0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)3/4, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)3/4, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)640, height=(int)480, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)640, height=(int)480, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstNvJpegEnc:nvjpegenc0.GstPad:src: caps = image/jpeg, sof-marker=(int)4, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)3/4, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
/GstPipeline:pipeline0/GstMultipartMux:multipartmux0.GstPad:sink_0: caps = image/jpeg, sof-marker=(int)4, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)3/4, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
/GstPipeline:pipeline0/GstMultipartMux:multipartmux0.GstPad:src: caps = multipart/x-mixed-replace, boundary=(string)ThisRandomString
/GstPipeline:pipeline0/GstMultipartDemux:multipartdemux0.GstPad:sink: caps = multipart/x-mixed-replace, boundary=(string)ThisRandomString
/GstPipeline:pipeline0/GstCapsFilter:capsfilter3: caps = image/jpeg, parsed=(boolean)true, width=(int)1280, height=(int)720, colorimetry=(string)2:4:7:1, framerate=(fraction)30/1, sof-marker=(int)0
/GstPipeline:pipeline0/GstCapsFilter:capsfilter3.GstPad:src: caps = image/jpeg, parsed=(boolean)true, width=(int)1280, height=(int)720, colorimetry=(string)2:4:7:1, framerate=(fraction)30/1, sof-marker=(int)0
/GstPipeline:pipeline0/GstV4l2Sink:v4l2sink0.GstPad:sink: caps = image/jpeg, parsed=(boolean)true, width=(int)1280, height=(int)720, colorimetry=(string)2:4:7:1, framerate=(fraction)30/1, sof-marker=(int)0
/GstPipeline:pipeline0/GstCapsFilter:capsfilter3.GstPad:sink: caps = image/jpeg
PosixMemMap:84 mmap failed : Cannot allocate memory
PosixMemMap:84 mmap failed : Cannot allocate memory
PosixMemMap:84 mmap failed : Cannot allocate memory
PosixMemMap:84 mmap failed : Cannot allocate memory

lead to the following kernel error:

[22640.875216] nvdec 15480000.nvdec: RISC-V booting from GSC
[22640.879175] nvdec 15480000.nvdec: RISCV boot success
[22672.238818] ptrace attach of "gst-launch-1.0 v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! nvjpegdec ! identity ! queue ! xvimagesink"[16754] was attempted by "gdb gst-launch-1.0 16754"[16761]
[24016.964050] BUG: non-zero pgtables_bytes on freeing mm: 12288