How to use nvivafilter gstreamer plugin on Xavier ?

I have been using nvivafilter gstreamer plugin with TX2 on various L4T releases up to R28.2.0.

However, on Xavier running R31.1, I am unable to get it working.

Even using the default version /usr/lib/aarch64-linux-gnu/libnvsample_cudaprocess.so, even without activating nor pre/post/cuda processing, it fails after 4 frames:

gst-launch-1.0 -v videotestsrc is-live=true ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! nvivafilter ! 'video/x-raw(memory:NVMM), format=RGBA' ! nvegltransform ! nveglglessink
Setting pipeline to PAUSED ...

Using winsys: x11 
Pipeline is live and does not need PREROLL ...
Got context from element 'eglglessink0': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/Gstnvivafilter:nvivafilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/GstNvEglTransform:nvegltransform0.GstPad:src: caps = video/x-raw(memory:EGLImage), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/GstEglGlesSink:eglglessink0.GstPad:sink: caps = video/x-raw(memory:EGLImage), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/GstNvEglTransform:nvegltransform0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/Gstnvivafilter:nvivafilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
nvbuf_utils: dmabuf_fd 1056 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
nvbuf_utils: dmabuf_fd 1056 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
streaming stopped, reason error (-5)
Execution ended after 0:00:00.200772524
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
nvbuf_utils: dmabuf_fd 1057 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
nvbuf_utils: dmabuf_fd 1058 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
nvbuf_utils: dmabuf_fd 1059 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
nvbuf_utils: dmabuf_fd 1056 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
Setting pipeline to NULL ...
XIO:  fatal IO error 9 (Bad file descriptor) on X server "�!�"
      after 18 requests (18 known processed) with 0 events remaining.

Using YUV420_SEMIPLANAR format doesn’t improve:

gst-launch-1.0 -v videotestsrc is-live=true ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! nvivafilter ! 'video/x-raw(memory:NVMM), format=NV12' ! nvoverlaysink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/Gstnvivafilter:nvivafilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12
/GstPipeline:pipeline0/GstNvOverlaySink-nvoverlaysink:nvoverlaysink-nvoverlaysink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12
/GstPipeline:pipeline0/Gstnvivafilter:nvivafilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)I420
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)320, height=(int)240, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
nvbuf_utils: dmabuf_fd 1030 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
nvbuf_utils: dmabuf_fd 1030 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
streaming stopped, reason error (-5)
Execution ended after 0:00:00.141727296
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
nvbuf_utils: dmabuf_fd 1031 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
nvbuf_utils: dmabuf_fd 1032 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
nvbuf_utils: dmabuf_fd 1033 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
nvbuf_utils: dmabuf_fd 1030 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
Setting pipeline to NULL ...
Freeing pipeline ...

Using NV12 or I420 as input doesn’t improve.

On a custom version of the library, I can see that the first four frames are processed, but then it get fooled. It happens out of the gpu_process (or any user defined function).

Is there a new way to use nvivafilter, or is it deprecated ?

Maybe related to videotestsrc…

I can get it working with camera:

gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM), format=NV12' ! nvivafilter cuda-process=true ! 'video/x-raw(memory:NVMM), format=RGBA' ! nvegltransform ! nveglglessink

but sometimes when the pipeline closes it triggers:

g_mutex_clear() called on uninitialised or locked mutex
Aborted (core dumped)

Furthermore, using I420 as input leads to similar dmabuf_fd error as with videotestsrc but ending up in seg fault:

gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM), format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! nvivafilter cuda-process=true ! 'video/x-raw(memory:NVMM), format=RGBA' ! nvegltransform ! nveglglessink
Setting pipeline to PAUSED ...

Using winsys: x11 
Pipeline is live and does not need PREROLL ...
Got context from element 'eglglessink0': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/Gstnvivafilter:nvivafilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)RGBA
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)RGBA
/GstPipeline:pipeline0/GstNvEglTransform:nvegltransform0.GstPad:src: caps = video/x-raw(memory:EGLImage), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)RGBA
/GstPipeline:pipeline0/GstEglGlesSink:eglglessink0.GstPad:sink: caps = video/x-raw(memory:EGLImage), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)RGBA
/GstPipeline:pipeline0/GstNvEglTransform:nvegltransform0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)RGBA
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)RGBA
/GstPipeline:pipeline0/Gstnvivafilter:nvivafilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :

GST_ARGUS: 2592 x 1944 FR = 29.999999 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 34000, max 550385000;

GST_ARGUS: 2592 x 1458 FR = 29.999999 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 34000, max 550385000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 22000, max 358733000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 1 
   Output Stream W = 2592 H = 1458 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd 1066 mapped entry NOT found
Caught SIGSEGV
#0  0x0000007f939be048 in __GI___poll (fds=0x55ae92af90, nfds=547938419120, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000007f93acbd80 in  () at /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#2  0x00000055ae6942d0 in  ()
Spinning.  Please run 'gdb gst-launch-1.0 9546' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:05.389870016
Setting pipeline to PAUSED ...
Setting pipeline to READY ...

Additional note: When some of these faults happened, the Ubuntu system catched these and proposed to report. I have accepted a few times, but it ended with /var/log/syslog to get almost 15GB and fulfilled my rootfs, so I’ve had to delete it…Don’t know what was the recurring error, but this might be easy to reproduce.

Hi HoneyP,
We can observe the issue. Will check and update.

Hi DaneLLL,
Any update info, patch or fix planned in next release about the above mentioned issues ?
Thanks

Hi HoneyP,
It shall be fixed in next release.

Hi Dane,

Nice to read that, thanks.
Is there a planned time window for next release ?

Hi Honey_Patouceul,

The next release will be published before Q2.

Thanks