Self-reboot when using Gstreamer pull video stream with rtmp

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
#1

Hi everybody,
We are doning some burning test on AGX devkit, my Jetson AGX Xavier has unexpected self-reboot several times when I was testing pull video stream from a server.
I use my NX pushing video stream to a server and AGX pulling the video stream from server.
Though they actually worked good at first several hours, AGX would self-reboot at sometime meanwhile NX still working good!

NX cmdline: “gst-launch-1.0 -v autovideosrc ! queue ! video/x-raw,width=640,height=480 ! tee name=t ! autovideoconvert ! nvv4l2h264enc ! h264parse ! flvmux name=mux ! queue ! rtmpsink location=rtmp://.../live/myStream alsasrc device=hw:3,0 ! audioconvert ! voaacenc ! queue ! mux. t. ! xvimagesink”

AGX cmdline: “gst-launch-1.0 -v rtmpsrc location=rtmp://.../live/myStream ! flvdemux name=demux ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! xvimagesink demux. ! queue ! aacparse ! faad ! audioconvert ! audioresample ! pulsesink”.

AGX kernel log is as follow:
kern.log (711.1 KB)
For example, unexpected reboot happens at Feb 11 00:08:30
Feb 11 00:08:30 agx kernel: [ 0.000000] Booting Linux on physical CPU 0x0
the details is in the attached log.

Any idea could this problem be?

Thanks
Jiaheng

Self-reboot when using Gstreamer pull video stream with rtmp
#2

HI, Jiaheng:
Could you also highligth the log which have unexpected reboot ?
Stephen

#3

Hi, stephen

The unexpected reboot happens at the Feb 11 00:08:30, which is highlighted as follows(the details can be found in the attached log),

Feb 10 22:42:20 agx kernel: [104571.130954] FAN rising trip_level:1 cur_temp:35000 trip_temps[2]:53000
Feb 10 22:57:17 agx kernel: [105468.205479] FAN cooling trip_level:0 cur_temp:25800 trip_temps[1]:35000
Feb 10 23:12:32 agx kernel: [106383.200928] FAN rising trip_level:1 cur_temp:35000 trip_temps[2]:53000
Feb 10 23:15:25 agx kernel: [106554.960632] bpmp: mrq 22 took 2532000 us
Feb 10 23:15:33 agx kernel: [106562.237698] bpmp: mrq 22 took 3116000 us
Feb 10 23:25:28 agx kernel: [107159.323576] FAN cooling trip_level:0 cur_temp:25850 trip_temps[1]:35000
Feb 10 23:41:23 agx kernel: [108114.642457] FAN rising trip_level:1 cur_temp:35000 trip_temps[2]:53000
Feb 10 23:55:28 agx kernel: [108959.087151] FAN cooling trip_level:0 cur_temp:25800 trip_temps[1]:35000

Feb 11 00:08:30 agx kernel: [ 0.000000] Booting Linux on physical CPU 0x0
Feb 11 00:08:30 agx kernel: [ 0.000000] Linux version 4.9.140-tegra (buildbrain@mobile-u64-4255) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) ) #1 SMP PREEMPT Fri Oct 16 12:25:00 PDT 2020
Feb 11 00:08:30 agx kernel: [ 0.000000] Boot CPU: AArch64 Processor [4e0f0040]
Feb 11 00:08:30 agx kernel: [ 0.000000] OF: fdt:memory scan node memory, reg size 48,
Feb 11 00:08:30 agx kernel: [ 0.000000] OF: fdt: - 80000000 , 2c000000
Feb 11 00:08:30 agx kernel: [ 0.000000] OF: fdt: - ac200000 , 44600000
Feb 11 00:08:30 agx kernel: [ 0.000000] OF: fdt: - 100000000 , 780000000
Feb 11 00:08:30 agx kernel: [ 0.000000] earlycon: tegra_comb_uart0 at MMIO32 0x000000000c168000 (options ‘’)
Feb 11 00:08:30 agx kernel: [ 0.000000] bootconsole [tegra_comb_uart0] enabled
Feb 11 00:08:30 agx kernel: [ 0.000000] Found tegra_fbmem: 00800000@a069d000
Feb 11 00:08:30 agx kernel: [ 0.000000] Found lut_mem: 00002008@a0698000

do you have any idea of the problem?

thanks
Jiaheng

#4

Moving to Jetson AGX Xavier forum for resolution.

#6

Hi,
Please run sudo tegrastats and check the thermal condition when the issue happens. Probably it is overheated.

#8

Hi,
We use 2 agx for pushing and pulling vedio stream with running sudo tegrastats.

agx1 is used to push stream and not reboot,
agx2 is used to pull stream and rebooted unexpectedly.

the rebooted agx’s log as follows, details can be found in attached log,

Feb 22 20:40:27 agx2-desktop kernel: [31082.605856] Task dump for CPU 0:
Feb 22 20:40:27 agx2-desktop kernel: [31082.605872] ksoftirqd/0 R running task 0 3 2 0x00000002
Feb 22 20:40:27 agx2-desktop kernel: [31082.605894] Call trace:
Feb 22 20:40:27 agx2-desktop kernel: [31082.605942] [] dump_backtrace+0x0/0x198
Feb 22 20:40:27 agx2-desktop kernel: [31082.605952] [] show_stack+0x24/0x30
Feb 22 20:40:27 agx2-desktop kernel: [31082.605970] [] sched_show_task+0xf8/0x148
Feb 22 20:40:27 agx2-desktop kernel: [31082.605979] [] dump_cpu_task+0x48/0x58
Feb 22 20:40:27 agx2-desktop kernel: [31082.605995] [] rcu_dump_cpu_stacks+0xb8/0xec
Feb 22 20:40:27 agx2-Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] Booting Linux on physical CPU 0x0
Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] Linux version 4.9.140-tegra (buildbrain@mobile-u64-4255) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) ) #1 SMP PREEMPT Fri Oct 16 12:25:00 PDT 2020
Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] Boot CPU: AArch64 Processor [4e0f0040]
Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] OF: fdt:memory scan node memory, reg size 48,
Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] OF: fdt: - 80000000 , 2c000000
Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] OF: fdt: - ac200000 , 44600000
Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] OF: fdt: - 100000000 , 780000000
Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] earlycon: tegra_comb_uart0 at MMIO32 0x000000000c168000 (options ‘’)
Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] bootconsole [tegra_comb_uart0] enabled
Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] Found tegra_fbmem: 00800000@a069d000
Feb 22 20:43:07 agx2-desktop kernel: [ 0.000000] Found lut_mem: 00002008@a0698000
tegrastats.txt (702.2 KB) kern.log (387.2 KB) kern_not_reboot.log (376.9 KB) tegrastats_not_reboot.txt (14.4 MB)

Thanks
Jiaheng

#9

Hi,
Please share the command on agx1 and agx2 so that we can set up to replicate the issue. And please provide your release version($ head -1 /etc/nv_tegra_release)

#10

Hi,
I use gstreamer cmdline like as follows,

// this is used to push stream (you may need a camera and rtmp server)
gst-launch-1.0 -v autovideosrc ! queue ! “video/x-raw,format=YUY2,width=640,height=480,framerate=30/1” ! autovideoconvert ! nvv4l2h264enc ! h264parse ! flvmux name=mux ! queue ! rtmpsink location=‘rtmp://10.13.1.70/live/myStream’ alsasrc ! audioconvert ! voaacenc ! queue ! mux.

// this is used to pull stream
gst-launch-1.0 rtmpsrc location=rtmp://10.13.1.70/live/myStream ! flvdemux name=demux ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! xvimagesink demux. ! queue ! aacparse ! faad ! audioconvert ! audioresample ! pulsesink

release version is :
# R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t186ref, EABI: aarch64, DATE: Fri Oct 16 19:37:08 UTC 2020

Thanks
Jiaheng

#11

Hi,
Could you share how to set up rtmp server? We run the command and hit the error:

$ gst-launch-1.0 -v nvarguscamerasrc ! nvv4l2h264enc ! h264parse ! flvmux name=mux ! queue ! rtmpsink location='rtmp://10.13.1.70/live/myStream' audiotestsrc ! audioconvert ! voaacenc ! queue ! mux.

ERROR: from element /GstPipeline:pipeline0/GstRTMPSink:rtmpsink0: Could not open resource for writing.
Additional debug info:
gstrtmpsink.c(245): gst_rtmp_sink_render (): /GstPipeline:pipeline0/GstRTMPSink:rtmpsink0:
Could not connect to RTMP stream "rtmp://10.13.1.70/live/myStream" for writing

We usually run RTSP and UDP. Don’t have much experience in RTMP. Please help provide guidance. Thanks.

#12

hi,
Maybe you can use my rtmp server at 1.15.126.186 (the bandwidth is very low, but you can still see the window)

So you may use pipeline like below ,

// push stream
gst-launch-1.0 -v autovideosrc ! queue ! video/x-raw,width=640,height=480 ! tee name=t ! autovideoconvert ! nvv4l2h264enc ! h264parse ! flvmux name=mux ! queue ! rtmpsink location=rtmp://1.15.126.86/live/myStream alsasrc ! audioconvert ! voaacenc ! queue ! mux. t. ! xvimagesink

// pull stream
gst-launch-1.0 -v rtmpsrc location=rtmp://1.15.126.86/live/myStream ! flvdemux name=demux ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! xvimagesink demux. ! queue ! aacparse ! faad ! audioconvert ! audioresample ! pulsesink

Thanks
Jiaheng

#13

Hi,
We are not able to make connection with rtmp://1.15.126.86/live/myStream. Could you check if it can be reproduced in RTSP? We have reference steps in
Jetson Nano FAQ
Q: Is there any example of running RTSP streaming?

#14

hi,

I will try to use RTSP on my agx.

And it is my typo that, you may try to this server address 1.15.126.186 with pipeline as follows,

// push stream
gst-launch-1.0 -v autovideosrc ! queue ! video/x-raw,width=640,height=480 ! tee name=t ! autovideoconvert ! nvv4l2h264enc ! h264parse ! flvmux name=mux ! queue ! rtmpsink location=rtmp://1.15.126.186/live/myStream alsasrc ! audioconvert ! voaacenc ! queue ! mux. t. ! xvimagesink

// pull stream
gst-launch-1.0 -v rtmpsrc location=rtmp://1.15.126.186/live/myStream ! flvdemux name=demux ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! xvimagesink demux. ! queue ! aacparse ! faad ! audioconvert ! audioresample ! pulsesink

Thanks
Jiaheng

#15

Hi DaneLLL,

I just upgrade my agx to
R32 (release), REVISION: 5.1, GCID: 26202423, BOARD: t186ref, EABI: aarch64, DATE: Fri Feb 19 16:50:29 UTC 2021 ,

and the good new is that the agx dose not reboot for these days. However, there is a problem that when use cmd “cheese”, and the agx report :
**(cheese:2466): Gtk-WARNING : 09:23:06.536: Theme parsing error: cheese.css:7:35: The style property GtkScrollbar:min-slider-length is deprecated and shouldn’t be used anymore. It will be removed in a future version
Opening in BLOCKING MODE
Segmentation fault (core dumped)

And I can still use pipeline to push video stream,
gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! "video/x-raw,format=YUY2,width=640,height=480,framerate=30/1" ! autovideoconvert ! nvv4l2h264enc ! h264parse ! flvmux name=mux ! queue ! rtmpsink location='rtmp://82.156.126.202/live/myStream1' alsasrc ! audioconvert ! voaacenc ! queue ! mux.

pull video stream
gst-launch-1.0 rtmpsrc location=rtmp://82.156.126.202/live/myStream1 ! flvdemux name=demux ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! xvimagesink demux. ! queue ! aacparse ! faad ! audioconvert ! audioresample ! pulsesink

The problem is I can only use v4l2src and can not use other video src element to enable my camera even the device is set to correct camera.

what is this problem?

Thanks
Jiaheng

#16

Hi,
The issue of failing to launch USB cameras through cheese or 3rdparty apps is a known issue on JP4.5. A similar topic:
VLC fails with L4T 32.5

It should work in gstreamer pipeline using v4l2src. Is it fine to use gstreamer? Or you have to use cheese app?

#17

Hi,
When I use agx to test push and pull video streams, the agx reboot again,
by the way,
# R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t186ref, EABI: aarch64, DATE: Fri Oct 16 19:37:08 UTC 2020

Log is
Mar 5 20:58:27 agx-desktop kernel: [ 25.435777] IPVS: Creating netns size=1920 id=7
Mar 5 20:58:27 agx-desktop kernel: [ 26.265424] IPVS: Creating netns size=1920 id=8
Mar 5 20:58:30 agx-desktop kernel: [ 28.476056] IPVS: Creating netns size=1920 id=9
Mar 5 20:58:33 agx-desktop kernel: [ 32.099552] ip6_tables: © 2000-2006 Netfilter Core Team
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00Mar 5 20:59:42 agx-desktop kernel: [ 0.000000] Booting Linux on physical CPU 0x0
Mar 5 20:59:42 agx-desktop kernel: [ 0.000000] Linux version 4.9.140-tegra (buildbrain@mobile-u64-4294) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) ) #1 SMP PREEMPT Tue Oct 27 21:02:46 PDT 2020
Mar 5 20:59:42 agx-desktop kernel: [ 0.000000] Boot CPU: AArch64 Processor [4e0f0040]
Mar 5 20:59:42 agx-desktop kernel: [ 0.000000] OF: fdt:memory scan node memory, reg size 48,
Mar 5 20:59:42 agx-desktop kernel: [ 0.000000] OF: fdt: - 80000000 , 2c000000
Mar 5 20:59:42 agx-desktop kernel: [ 0.000000] OF: fdt: - ac200000 , 44600000
Mar 5 20:59:42 agx-desktop kernel: [ 0.000000] OF: fdt: - 100000000 , 780000000
Mar 5 20:59:42 agx-desktop kernel: [ 0.000000] earlycon: tegra_comb_uart0 at MMIO32
kern.log.1 (8.6 MB)

Is there any problem with # R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t186ref, EABI: aarch64, DATE: Fri Oct 16 19:37:08 UTC 2020 ?

Thanks
Jiaheng

#18

Hi,
Please apply this prebuilt lib to r32.5.1 and check if you can run cheese app successfully:
Libv4l2_nvargus.so source code - #19 by JerryChang