GStreamer lockup with H.264 encoder from nvarguscamerasrc

Autonomous Machines Jetson & Embedded Systems Jetson TX2
,
17

Yes, I have reproduced the issue with both fakesink (see post #10) and with udpsink to the localhost as such.

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! udpsink host=localhost port=5000 sync=false async=false
gst-launch-1.0 -e nvarguscamerasrc sensor-id=1 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! udpsink host=localhost port=5002 sync=false async=false
gst-launch-1.0 -e nvarguscamerasrc sensor-id=2 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! udpsink host=localhost port=5004 sync=false async=false
gst-launch-1.0 -e nvarguscamerasrc sensor-id=3 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! udpsink host=localhost port=5006 sync=false async=false
gst-launch-1.0 -e nvarguscamerasrc sensor-id=4 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! udpsink host=localhost port=5008 sync=false async=false
gst-launch-1.0 -e nvarguscamerasrc sensor-id=5 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! udpsink host=localhost port=5010 sync=false async=false

With nvv4l2h264enc and udpsink to the localhost:

  • Camera 5 locked up in 7m34s
  • Camera 1 locked up in 17m10s
  • Camera 3 locked up in 19m5s
  • Camera 4 locked up in 1h32m
  • Camera 0 locked up in 4h59m
  • Camera 2 locked up in 9h50m

No, h264parse does not help. I combined h264parse and fakesink as such:

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=1 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=2 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=3 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=4 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=5 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink

With h264parse added between nvv4l2h264enc and rtph264pay, and with fakesink:

  • Camera 5 locked up in 31s
  • Camera 1 locked up in 2m40s
  • Camera 4 locked up in 3m3s
  • Camera 2 locked up in 1h2m
  • Camera 3 locked up in 2h1m
  • Camera 0 locked up in 2h18m

Yes, when I launch the 6 pipelines from the same gst-launch-1.0 process the results are the same. I ran the following combined gst-launch-1.0 command:

GST_DEBUG="*:6" GST_DEBUG_NO_COLOR=1 gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink nvarguscamerasrc sensor-id=1 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink nvarguscamerasrc sensor-id=2 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink nvarguscamerasrc sensor-id=3 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink nvarguscamerasrc sensor-id=4 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink nvarguscamerasrc sensor-id=5 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! h264parse ! rtph264pay mtu=1400 ! fakesink

This was more difficult to determine when exactly each GStreamer pipeline had locked up. Looking at htop you can see the top six threads of gst-launch-1.0 with significant execution time accumulation are all at 0% CPU usage:

htop-gst-launch-locked-up
htop-gst-launch-locked-up736×461 32.7 KB

At this point the NVENC clock frequency has gone from 1164MHz to OFF.

This is the output from the gst-launch-1.0 debug output when GStreamer was locked up:
gstreamer-all-camera-stream-lockup.txt

Here is the camera VI tracing log from debugfs with ~10 seconds duration when GStreamer was locked up:
camera-all-channels-streaming-trace.txt

Finally, here is my little bash command to pull out the interesting information from the trace log:

$ cat camera-all-channels-streaming-trace.txt | cut -d' ' -f 13-14 | sort | uniq
tag:ATOMP_FE channel:0x00
tag:ATOMP_FE channel:0x01
tag:ATOMP_FE channel:0x02
tag:ATOMP_FE channel:0x03
tag:ATOMP_FE channel:0x04
tag:ATOMP_FE channel:0x05
tag:ATOMP_FS channel:0x00
tag:ATOMP_FS channel:0x01
tag:ATOMP_FS channel:0x02
tag:ATOMP_FS channel:0x03
tag:ATOMP_FS channel:0x04
tag:ATOMP_FS channel:0x05
tag:CHANSEL_LOAD_FRAMED channel:0x01
tag:CHANSEL_LOAD_FRAMED channel:0x04
tag:CHANSEL_LOAD_FRAMED channel:0x10
tag:CHANSEL_LOAD_FRAMED channel:0x41
tag:CHANSEL_LOAD_FRAMED channel:0x44
tag:CHANSEL_LOAD_FRAMED channel:0x50
tag:CHANSEL_PXL_EOF channel:0x00
tag:CHANSEL_PXL_EOF channel:0x01
tag:CHANSEL_PXL_EOF channel:0x02
tag:CHANSEL_PXL_EOF channel:0x03
tag:CHANSEL_PXL_EOF channel:0x04
tag:CHANSEL_PXL_EOF channel:0x05
tag:CHANSEL_PXL_SOF channel:0x00
tag:CHANSEL_PXL_SOF channel:0x01
tag:CHANSEL_PXL_SOF channel:0x02
tag:CHANSEL_PXL_SOF channel:0x03
tag:CHANSEL_PXL_SOF channel:0x04
tag:CHANSEL_PXL_SOF channel:0x05

So it looks like all six(6) channels are still receiving valid CSI frame data. My low-level hardware driver also indicates that CSI frames are still being sent to the NVIDIA CSI ports, even though GStreamer has locked up.

I do not have the exact lock up times of each of those individual GStreamer pipelines, but based on the final output from gstreamer-all-camera-stream-lockup.txt you can see the final lock up occurred at 16h43m. I remember seeing as it was running, those high utilization gst-launch-1.0 threads started to go to <1.0% CPU utilization. I interpret that to mean that the GStreamer pipeline was gradually locking up for each video stream one-by-one just like before. At the end the NVENC clock frequency was set to OFF when all the GStreamer debug output had halted and things were locked up.

I did two things. First, I ran the following commands to boost the camera VI/NVCSI/ISP clocks during the test:

echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
export GST_DEBUG=“*:6”
export GST_DEBUG_NO_COLOR=1
gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=1 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=2 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=3 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=4 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=5 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink

With boosted camera VI/NVCSI/ISP clocks:

  • Camera 5 locked up in 2m48s
  • Camera 3 locked up in 13m55s
  • Camera 4 locked up in 26m24s
  • Camera 1 locked up in 1h23m
  • Camera 2 locked up in 4h49m
  • Camera 0 locked up in 6h45m

Second, I also followed the Xavier link to change the VIC to the userspace governor with the maximum frequency:

echo on > /sys/devices/13e10000.host1x/15340000.vic/power/control
echo userspace > /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/governor
echo 1024000000 > /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/max_freq
echo 1024000000 > /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/userspace/set_freq
export GST_DEBUG=“*:6”
export GST_DEBUG_NO_COLOR=1
gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=1 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=2 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=3 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=4 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink
gst-launch-1.0 -e nvarguscamerasrc sensor-id=5 sensor-mode=0 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! fakesink

With VIC userspace governor and maximum clock frequency of 1024000000 (1024 MHz):

  • Camera 4 locked up in 48s
  • Camera 5 locked up in 14m59s
  • Camera 0 locked up in 47m5s
  • Camera 2 locked up in 3h16m
  • Camera 3 locked up in 5h11m
  • Camera 1 locked up in 7h31m