Processing NV12 buffer from nvarguscamerasrc

[arbalasquid]

Hello,

I have a gstreamer pipeline that grabs video raw NV12 images from my cameras using an orin and the nvarguscamerasrc plugin. I have C++ code that setsup the pipeline and I’m able to get video to output h264 packets correctly, however, I need to also do some processing to the image before encoding. Said processing requires me to operate on the buffer that comes out of nvarguscamerasrc before encoding.

I have a gst_pad_add_probe call to the pad for the nvarguscamerasrc and I’m able to get calls to my callback, however, when I look into the GstMapInfo object size, its 64 bytes, but I expect a way bigger size since its a 1080p image. The video comes through the udpsink just fine as a h264 encoded stream and I’m able to view the video on the other side, but still confused why the buffers are not the full video frames.

Any help or pointers would be appreciated, thanks in advance!

[JerryChang]

hello arbalasquid,

may I know what’s your input camera formats?
you may refer to below gst pipeline for running camera preview stream and also video recording.
$ gst-launch-1.0 nvarguscamerasrc num-buffers=300 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' ! tee name=streams streams. ! queue ! nvv4l2h265enc bitrate=8000000 ! h265parse ! qtmux ! filesink location=h265.mp4 streams. ! queue ! nvvidconv ! xvimagesink -e

[arbalasquid]

Hi, thanks for the response. The camera outputs 1920x1080 NV12 frames. Here is the pipeline that I’m using:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ee-mode=0 do-timestamp=1 ! “video/x-raw(memory:NVMM), format=NV12, framerate=30/1” ! queue ! nvvidconv ! “video/x-raw(memory:NVMM), format=NV12” ! nvv4l2h264enc insert-sps-pps=true bitrate=30000000 preset-level=2 control-rate=1 maxperf-enable=1 ratecontrol-enable=0 bufapi-version=0 ! “video/x-h264,stream-format=byte-stream” ! queue ! rtph264pay ssrc=335827 ! udpsink host=192.168.2.10 port=50010 sync=false async=false ttl=32

On the receiving UDP end, the images come out fine, but when I setup a pad callback for the src of queue, and even nvarguscamerasrc, the buffers are not the format I expect. Upon checking the caps of the pad, it is indeed NV12 1920x1080 format, but the buffer sizes are only 64 bytes.

[arbalasquid]

Kinda solved, since I assume the 64bytes is not the actual frame, I assumed the actual data has to be in a DMA buffer and since there was no easy way to map from DMA, I just added an nvvidconv to convert to x-video/raw, format=BGRx to do my operations with OpenCV and then another nvvidconv to go back to NV12 x-video/raw(memory:NVMM), format=NV12

This allowed me to actually get the data and do some cpu operations on the buffer.