HW encoding and decoding speed

[tmirzq]

I have a Basler camera connected to a Jetson AGX Orin development kit. I want to send the image from the Jetson to my PC, preprocess it there (decoding it first, then encoding it again after processing), and then send the processed image back to the Jetson for final decoding. The HW encoding and decoding on the Jetson works oke, but the HW nvidia encoding and decoding on my PC is awful. NVIDIA encoding and decoding adds around 350 to 400ms latency to my pipeline, compared to SW openh264 encoding and decoding. Somewhere I am doing something wrong, but I don’t get it really. For the video streaming I am using the following commands:

Jetson (encoding Basler image and sending over UDP to my PC):

gst-launch-1.0 pylonsrc ! ‘video/x-raw,framerate=90/1,format=GRAY8’ ! nvvideoconvert ! ‘video/x-raw(memory:NVMM), width=1920,height=1080,format=NV12’ ! nvv4l2h264enc bitrate=4000000 ! h264parse ! rtph264pay config-interval=1 ! queue ! udpsink host=192.168.2.236 port=8554 -e

PC (nvidia encoding and decoding, container with deepstream 7.1 version, with nvh264enc and nvh264dec results are similar):

gst-launch-1.0 udpsrc port=8554 caps=“application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96” ! rtph264depay ! h264parse ! nvv4l2decoder ! nvv4l2h264enc ! rtph264pay ! queue ! udpsink host=192.168.2.237 port=8888 -e

Jetson (decoding):

gst-launch-1.0 udpsrc port=8888 caps=“application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96” ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvideoconvert ! autovideosink sync=false

With these I get latency around 500ms, but changing the PC command t oopenh264, the latency drops to around 120ms. Clearly the bottleneck is in the PC, but how to correct it? When I start the nvidia encoding and decoding on PC, nvidia-smi shows that gst uses around 250 GPU MBs, so from my understanding it works, but somehow I am clearly making a mistake in the memory managment. This is the command with openh264:

gst-launch-1.0 udpsrc port=8554 caps=“application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96” ! rtph264depay ! h264parse ! openh264dec ! openh264enc bitrate=4000000 complexity=0 rate-control=1 ! rtph264pay ! udpsink host=192.168.2.237 port=8888 -e

This is the nvidia-smi output:

[DaneLLL]

Hi,
Enabling the properties shall have latency enhancement:

// nvv4l2h264enc (poc-type=2)
  maxperf-enable      : Enable or Disable Max Performance mode
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  poc-type            : Set Picture Order Count type value
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 2 Default: 0
// nvv4l2decoder
  disable-dpb         : Set to disable DPB buffer for low latency
                        flags: readable, writable
                        Boolean. Default: false
  enable-max-performance: Set to enable max performance
                        flags: readable, writable
                        Boolean. Default: false

Please enable these and check.