Can nv3dsink get keyboard events?

We now have such a demand, we want to display the lens image data through HDMI, and then modify the degree of image distortion through the keyboard, the distortion algorithm is implemented in the nvivafilter plug-in.

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 wbmode=0  saturation=0 blocksize=24834048 ! "video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008, format=(string)NV12, framerate=(fraction)29/1" ! tee name=t t. !  queue ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12' !  nvivafilter cuda-process=true customer-lib-name="libnvsample_cudaprocess.so" ! nv3dsink 

hello 1508723374,

no, nv3dsink it only has capability to parse window coordinate (x, y) and resolution (width, height).

In the process of research, I found that xvimagesink plug-in has the ability to capture keyboard events, and this plug-in can also send events up. I need to get the keyboard event from xvimagesink in the nvivafilter plugin, what should I do?

Hi,
You may check the source code of xvimagesink to know how it gets keyboard event, and apply it to callback function of nvivafilter plugin.

Also for your reference, we have samples of reading keyboard events in

Jetson Linux 36.3 | NVIDIA Developer
Driver Package (BSP) Sources
nvgstapps_src.tbz2

Thank you very much for your reply, I believe the code you gave me is helpful, but I have not progressed to loading the code. During the debugging process I found that the nvivafilter plugin did not have a way to combine with xvimagesink. In time nothing is done in nvsample_cudaprocess.cu and there is no way to combine it.

Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 4128 x 3008 FR = 28.999999 fps Duration = 34482760 ; Analog Gain range min 1.000000, max 251.188705; Exposure Range min 11000, max 660000000;
GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 4128 H = 3008
seconds to Run = 0
Frame Rate = 28.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
Acquired Frame: 10, time sec 0 msec 900 id:281473164767712
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0:
streaming stopped, reason not-negotiated (-4)
EOS on shutdown enabled – waiting for EOS after Error
Waiting for EOS…

This is a gst command that does not work properly.

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 wbmode=0  saturation=0 blocksize=24834048 ! "video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008, format=(string)NV12, framerate=(fraction)29/1" ! nvivafilter post-process=true customer-lib-name="libnvsample_cudaprocess.so" ! nvvidconv ! videoconvert  ! ximagesink

But it worked fine when I used the nv3dsink plugin.

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 wbmode=0  saturation=0 blocksize=24834048 ! "video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008, format=(string)NV12, framerate=(fraction)29/1" !  nvivafilter cuda-process=true customer-lib-name="libnvsample_cudaprocess.so" ! nvvidconv !  nv3dsink

Below is my source code, I hope you can provide me with some help.
nvsample_cudaprocess.zip (4.0 KB)
At the same time, I have encountered similar problems in other posts, but it doesn’t seem to have helped me in any particular way.
Using VPI in GStreamer - Jetson & Embedded Systems / Jetson AGX Orin - NVIDIA Developer Forums

My platform version is as follows

hl@hl-desktop:~/work/nvsample_gst_cudaprocess$ cat /etc/nv_tegra_release 
# R35 (release), REVISION: 3.1, GCID: 32827747, BOARD: t186ref, EABI: aarch64, DATE: Sun Mar 19 15:19:21 UTC 2023

Hi,
You may break down the pipeline. Can try

$ gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 wbmode=0  saturation=0 blocksize=24834048 ! "video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008, format=(string)NV12, framerate=(fraction)29/1" ! nvivafilter post-process=true customer-lib-name="libnvsample_cudaprocess.so" ! nvvidconv ! video/x-raw,format=I420 ! fakesink

If it works, you can then try

$ gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 wbmode=0  saturation=0 blocksize=24834048 ! "video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008, format=(string)NV12, framerate=(fraction)29/1" ! nvivafilter post-process=true customer-lib-name="libnvsample_cudaprocess.so" ! nvvidconv ! video/x-raw,format=I420 ! xvimagesink

This doesn’t work.

Hi,
You may try

$ gst-launch-1.0 nvarguscamerasrc ! nvivafilter post-process=true customer-lib-name="libnvsample_cudaprocess.so" ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! video/x-raw,format=I420 ! fakesink

Still not able to work, error as follows.

pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 4128 x 3008 FR = 28.999999 fps Duration = 34482760 ; Analog Gain range min 1.000000, max 251.188705; Exposure Range min 11000, max 660000000;
ARGUS_ERROR: Error generated. gstnvarguscamerasrc.cpp, execute: 918 Frame Rate specified is greater than supported
GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 4128 H = 3008
seconds to Run = 0
Frame Rate = 28.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
ARGUS_ERROR: Error generated. gstnvarguscamerasrc.cpp, execute: 1154 InvalidState.
GST_ARGUS: Cleaning up
xxxx3
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
NvBufSurfaceFromFd Failed.Execution ended after 0:00:00.040295786
Setting pipeline to NULL …
CONSUMER: ERROR OCCURRED
Freeing pipeline …

Thank you very much for your reply. The command you replied is of great reference significance to me. I have modified the following command to work.I’ll focus on the original keyboard event.

gst-launch-1.0 nvarguscamerasrc sensor-id=0 wbmode=0  saturation=0 blocksize=24834048 ! "video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008, format=(string)NV12, framerate=(fraction)29/1" ! nvivafilter post-process=true customer-lib-name="libnvsample_cudaprocess.so" ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! video/x-raw,format=I420 ! xvimagesink

I have now returned to the original keyboard event itself. From the code you sent me, getting the keyboard event just starts a thread and uses getchar to read the keyboard. We have thought of this method before, but it does not meet our actual requirements, we prefer to use the xvimagesink event form to complete the task. Because it’s not just keyboard data, mouse events may be used later.
In xvimagesink, the event type GST_EVENT_NAVIGATION fits our requirements just fine.