Performance of an analytics pipeline based on GRAY8 frames

[valentin.savchenko]

Hi,

I have an app which receives GRAY8 frames from an industrial camera. To be able to feed the frames to nvstreammux and eventually inference on them with nvinfer, one have to convert the frames into NV12/RGBA buffers.

Experiments with gst-launch-1.0 showed that conversion from GRAY8 to any color option impacts overall FPS of an analytics pipeline most dramatically. Couple examples:

gst-launch-1.0 -e -v multifilesrc location=/home/data/cars/materials/Russia/Moscow_M1__1/photo/2022-03-09_14-14-38_rec/frame_%05d.jpg start-index=0 stop-index=-1 caps=image/jpeg,framerate=(fraction)50/1 ! jpegdec ! videoconvert ! video/x-raw,format=GRAY8 ! fpsdisplaysink text-overlay=false video-sink=fakesink sync=false

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 551, dropped: 0, current: 21,00, average: 21,04

gst-launch-1.0 -e -v multifilesrc location=/home/data/cars/materials/Russia/Moscow_M1__1/photo/2022-03-09_14-14-38_rec/frame_%05d.jpg start-index=0 stop-index=-1 caps=image/jpeg,framerate=(fraction)50/1 ! jpegdec ! videoconvert ! video/x-raw,format=GRAY8 ! videoconvert ! video/x-raw,format=BGRx ! fpsdisplaysink text-overlay=false video-sink=fakesink sync=false

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 709, dropped: 0, current: 5,89, average: 5,88

gst-launch-1.0 -e -v multifilesrc location=/home/data/cars/materials/Russia/Moscow_M1__1/photo/2022-03-09_14-14-38_rec/frame_%05d.jpg start-index=0 stop-index=-1 caps=image/jpeg,framerate=(fraction)50/1 ! jpegdec ! videoconvert ! video/x-raw,format=GRAY8 ! videoconvert ! video/x-raw,format=BGRx ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=RGBA’ ! m.sink_0 nvstreammux name=m batch-size=1 width=2448 height=2048 ! nvinfer name=nvinfer_cd config-file-path=/home/data/valsav/vision/data/config/deepstream/jetson-tx2/cd/pgie-1.txt ! nvtracker tracker-width=640 tracker-height=384 gpu-id=0 enable-batch-process=1 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvdcf.so ll-config-file=/home/data/valsav/vision/data/config/deepstream/jetson-tx2/cd/tracker_config.yml ! nvinfer name=nvinfer_lpd config-file-path=/home/data/valsav/vision/data/config/deepstream/jetson-tx2/lpd/sgie-2_lpd.txt ! nvinfer name=nvinfer_lpcc config-file-path=/home/data/valsav/vision/data/config/deepstream/jetson-tx2/lpcc/sgie-4_lpcc.txt ! nvinfer name=nvinfer_lpr config-file-path=/home/data/valsav/vision/data/config/deepstream/jetson-tx2/lpr/sgie-3_lpr.txt ! fpsdisplaysink text-overlay=false video-sink=fakesink sync=false

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1312, dropped: 0, current: 5,35, average: 5,32

Furthermore, it appears that among all color options NV12 allows to achieve the best FPS:

gst-launch-1.0 -e -v multifilesrc location=/home/data/cars/materials/Russia/Moscow_M1__1/photo/2022-03-09_14-14-38_rec/frame_%05d.jpg start-index=0 stop-index=-1 caps=image/jpeg,framerate=(fraction)50/1 ! jpegdec ! videoconvert ! video/x-raw,format=GRAY8 ! videoconvert ! video/x-raw,format=NV12 ! fpsdisplaysink text-overlay=false video-sink=fakesink sync=false

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 285, dropped: 0, current: 7,87, average: 7,83

gst-launch-1.0 -e -v multifilesrc location=/home/data/cars/materials/Russia/Moscow_M1__1/photo/2022-03-09_14-14-38_rec/frame_%05d.jpg start-index=0 stop-index=-1 caps=image/jpeg,framerate=(fraction)50/1 ! jpegdec ! videoconvert ! video/x-raw,format=GRAY8 ! videoconvert ! video/x-raw,format=NV12 ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=NV12’ ! m.sink_0 nvstreammux name=m batch-size=1 width=2448 height=2048 ! nvinfer name=nvinfer_cd config-file-path=/home/data/valsav/vision/data/config/deepstream/jetson-tx2/cd/pgie-1.txt ! nvtracker tracker-width=640 tracker-height=384 gpu-id=0 enable-batch-process=1 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvdcf.so ll-config-file=/home/data/valsav/vision/data/config/deepstream/jetson-tx2/cd/tracker_config.yml ! nvinfer name=nvinfer_lpd config-file-path=/home/data/valsav/vision/data/config/deepstream/jetson-tx2/lpd/sgie-2_lpd.txt ! nvinfer name=nvinfer_lpcc config-file-path=/home/data/valsav/vision/data/config/deepstream/jetson-tx2/lpcc/sgie-4_lpcc.txt ! nvinfer name=nvinfer_lpr config-file-path=/home/data/valsav/vision/data/config/deepstream/jetson-tx2/lpr/sgie-3_lpr.txt ! fpsdisplaysink text-overlay=false video-sink=fakesink sync=false

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 237, dropped: 0, current: 7,50, average: 7,51

Hence the question, is there any options to further speed up a pipeline that starts with GRAY8 frames at input?

My setup:

• Jetson TX2, 8GB
• L4T R32.5.0
• DeepStream 5.1

Valentin

[DaneLLL]

Hi,
Please try the command and check if it can achieve target frame rate:

gst-launch-1.0 -e -v multifilesrc location=/home/data/cars/materials/Russia/Moscow_M1__1/photo/2022-03-09_14-14-38_rec/frame_%05d.jpg start-index=0 stop-index=-1 caps=image/jpeg,framerate=(fraction)50/1 ! jpegdec ! nvvideoconvert ! video/x-raw(memory:NVMM),format=NV12 ! ! fpsdisplaysink text-overlay=false video-sink=fakesink sync=false

[valentin.savchenko]

Hi,

There is a problem with direct conversion from GRAY8/RAW to NV12/NVMM with nvvideoconvert. Let me illustrate it. The baseline:

gst-launch-1.0 -e -v videotestsrc ! ‘video/x-raw,width=2448,height=2048,format=GRAY8,framerate=50/1’ ! nvjpegenc quality=95 ! multifilesink location=frame_%05d.jpg

and with nvvideoconvert:

gst-launch-1.0 -e -v videotestsrc ! ‘video/x-raw,width=2448,height=2048,format=GRAY8,framerate=50/1’ ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=NV12’ ! nvvideoconvert ! ‘video/x-raw,format=GRAY8’ ! nvjpegenc quality=95 ! multifilesink location=frame_%05d.jpg

As a result inference on such input works substantially poorer, so the question is:

• Is there any parameters for nvvideoconvert which would prevent it from such dramatic quantization of GRAY8 images?

Valentin

[DaneLLL]

Hi,
Please try this solution:
Green tint video - #15 by DaneLLL

Set to bt601 in converting to I420/NV12.