Pipeline freeze?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) Jetson Xavier AGX
• DeepStream Version 6.3.0
• JetPack Version (valid for Jetson only) 5.1
• TensorRT Version 8.5.2.2

I am facing a pipeline freeze that I can’t understand. My sample code is deepstream_python_apps/apps/deepstream-test3/deepstream_test_3.py at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub. I am using a Yolo model, but it is still an issue if you choose the traditional resnet18_trafficcamnet.

I start the script deepstream_test_3.py with 4 RTSP sources of the same camera, producing an mosaic of 4 videos at the nv3dsink. This is at the top-left of the following image. Almost at the same time I started nvgstplayer to watch the same stream (now a 5th one). It is at the bottom-right of the image.

Screenshot from 2024-05-11 14-24-00
Screenshot from 2024-05-11 14-24-001920×1080 281 KB

After some minutes, like 30-40 minutes, the nv3dsink show all images freezed. At first I thought the camera stopped providing images, and this was the reason I am using nvstrplayer as a counterproof, it is still showing the moving images. So I concluded that the images are being produced and arriving at the uridecodebin.

Then I tried to inspect the pipeline. I created a dummy buffer prove, with nothing more than this. I attached it to the source pad from the nvvideoconvert.

def dummy_buffer_probe(pad,info,u_data):
print(“Buffer Probe”)

Whenever the images from nv3dsink freeze, the buffer probe stops printing “Buffer Probe”. But look at the console output (bottom-left), the nvdslogger says it is running at about 12fps, so there must be something in the pipeline. I have even moved the buffer probe to the source pad of the pgie nvinfer and the same happens.

So, what may be happening?

3

Does it happen every time and how long does it happen after you run the pipeline?
Could you use gdb tool to run the command and check the thread info when it freezes?

$gdb --args <your command>
$r
After it hangs, use the command to check the thread info
$ info threads
$ thread <thread_id>
4

It happens 100% of the runs, but the time necessary to it happen I can’t precise, sometimes it takes 40 minutes, sometimes 3 hours.

I have never used gdb before, so I am quite blind on what I am doing. I waited for the freeze, did the Ctrl-C and did info threads, but I can1t read it.

This is what suddenly appear in terminal before Ctrl-C:

> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> [Thread 0xfffe9a7fc0b0 (LWP 1214948) exited]
> nvstreammux: Successfully handled EOS for source_id=3
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> Buffer Probe
> [Thread 0xfffe51a040b0 (LWP 1214970) exited]
> **PERF : FPS_0 (8.09)	FPS_1 (0.00)	FPS_2 (9.60)	FPS_3 (0.00)	
> Warning: gst-resource-error-quark: Could not read from resource. (9): gstrtspsrc.c(5624): gst_rtspsrc_loop_udp (): /GstPipeline:pipeline0/GstBin:source-bin-01/GstURIDecodeBin:uri-decode-bin/GstRTSPSrc:source:
> The server closed the connection.
> Warning: gst-resource-error-quark: Could not read from resource. (9): gstrtspsrc.c(5624): gst_rtspsrc_loop_udp (): /GstPipeline:pipeline0/GstBin:source-bin-00/GstURIDecodeBin:uri-decode-bin/GstRTSPSrc:source:
> The server closed the connection.
> Warning: gst-resource-error-quark: Could not read from resource. (9): gstrtspsrc.c(5624): gst_rtspsrc_loop_udp (): /GstPipeline:pipeline0/GstBin:source-bin-03/GstURIDecodeBin:uri-decode-bin/GstRTSPSrc:source:
> The server closed the connection.
> Warning: gst-resource-error-quark: Could not read from resource. (9): gstrtspsrc.c(5624): gst_rtspsrc_loop_udp (): /GstPipeline:pipeline0/GstBin:source-bin-02/GstURIDecodeBin:uri-decode-bin/GstRTSPSrc:source:
> The server closed the connection.
> **PERF : FPS_0 (8.08)	FPS_1 (0.00)	FPS_2 (9.59)	FPS_3 (0.00)

And this is what I get after Ctrl-C + info thread:

  Id   Target Id                                             Frame 
* 1    Thread 0xfffff61d3010 (LWP 1214865) "python3"         0x0000fffff7e95098 in __GI___poll (fds=0x117c4910, nfds=3, timeout=<optimized out>)
    at ../sysdeps/unix/sysv/linux/poll.c:41
  2    Thread 0xfffff34f70b0 (LWP 1214875) "python3"         0x0000fffff7e977b0 in __GI___select (nfds=<optimized out>, readfds=0xfffff34e95c0, 
    writefds=0xfffff34e9540, exceptfds=0xfffff34e94c0, timeout=0xfffff34e94b0) at ../sysdeps/unix/sysv/linux/select.c:53
  3    Thread 0xffffe46220b0 (LWP 1214881) "python3"         futex_abstimed_wait_cancelable (private=0, abstime=0xffffe46216c8, 
    clockid=<optimized out>, expected=0, futex_word=0xfffff2b36400) at ../sysdeps/nptl/futex-internal.h:320
  4    Thread 0xffff9f02f0b0 (LWP 1214882) "python3"         syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  5    Thread 0xffff9e5f00b0 (LWP 1214883) "cuda-EvtHandlr"  0x0000fffff7e95098 in __GI___poll (fds=0xffff90000bc0, nfds=9, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  6    Thread 0xffff9ddef0b0 (LWP 1214884) "python3"         futex_wait_cancelable (private=0, expected=0, futex_word=0xf51038)
    at ../sysdeps/nptl/futex-internal.h:183
  7    Thread 0xffff9cb640b0 (LWP 1214885) "python3"         futex_wait_cancelable (private=0, expected=0, futex_word=0x1e3317c)
    at ../sysdeps/nptl/futex-internal.h:183
  8    Thread 0xffff7bc970b0 (LWP 1214886) "python3"         futex_wait_cancelable (private=0, expected=0, futex_word=0x1e332d4)
    at ../sysdeps/nptl/futex-internal.h:183
  9    Thread 0xffff7979e0b0 (LWP 1214887) "python3"         syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  10   Thread 0xffff78f9d0b0 (LWP 1214888) "python3"         syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  11   Thread 0xffff728600b0 (LWP 1214889) "python3"         futex_wait_cancelable (private=0, expected=0, futex_word=0xe8f5828)
    at ../sysdeps/nptl/futex-internal.h:183
  12   Thread 0xffff7205f0b0 (LWP 1214890) "task0"           0x0000fffff7e95098 in __GI___poll (fds=0xffff7205d2a8, nfds=2, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  13   Thread 0xffff7185e0b0 (LWP 1214891) "task1"           0x0000fffff7e95098 in __GI___poll (fds=0xffff7185c2a8, nfds=2, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  14   Thread 0xffff7105d0b0 (LWP 1214892) "task2"           0x0000fffff7e95098 in __GI___poll (fds=0xffff7105b2a8, nfds=2, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  15   Thread 0xffff7085c0b0 (LWP 1214893) "task3"           0x0000fffff7e95098 in __GI___poll (fds=0xffff7085a2a8, nfds=2, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  16   Thread 0xffff5bfff0b0 (LWP 1214894) "queue14:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  17   Thread 0xffff5b7fe0b0 (LWP 1214895) "queue13:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  18   Thread 0xffff5affd0b0 (LWP 1214896) "queue12:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  19   Thread 0xffff5a7fc0b0 (LWP 1214897) "queue2:src"      syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  20   Thread 0xffff59ffb0b0 (LWP 1214898) "queue11:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  21   Thread 0xffff597fa0b0 (LWP 1214899) "queue1:src"      syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  22   Thread 0xffff58ff90b0 (LWP 1214900) "queue10:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  23   Thread 0xffff37fff0b0 (LWP 1214901) "gmain"           0x0000fffff7e95098 in __GI___poll (fds=0xffff680131f0, nfds=1, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  24   Thread 0xffff377fe0b0 (LWP 1214902) "dconf worker"    0x0000fffff7e95098 in __GI___poll (fds=0xffff68016040, nfds=1, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  25   Thread 0xffff36ffd0b0 (LWP 1214903) "gdbus"           0x0000fffff7e95098 in __GI___poll (fds=0xffff28012500, nfds=2, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  26   Thread 0xffff367fc0b0 (LWP 1214904) "fakesrc0:src"    syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  27   Thread 0xffff35ffb0b0 (LWP 1214905) "fakesrc1:src"    syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  28   Thread 0xffff357fa0b0 (LWP 1214906) "fakesrc2:src"    syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  29   Thread 0xffff34ff90b0 (LWP 1214907) "rtpsession-rtcp" __lll_lock_wait (futex=futex@entry=0xffff68016000, private=0) at lowlevellock.c:52
  30   Thread 0xffff17fff0b0 (LWP 1214908) "udpsrc0:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  31   Thread 0xffff177fe0b0 (LWP 1214909) "udpsrc1:src"     0x0000fffff7e95098 in __GI___poll (fds=0xffff177fd268, nfds=2, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  32   Thread 0xffff16ffd0b0 (LWP 1214910) "rtpsession-rtcp" 0x0000fffff7e951a4 in __ppoll (fds=0xffff540164a0, nfds=1, timeout=<optimized out>, 
    sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44
  33   Thread 0xffff167fc0b0 (LWP 1214911) "udpsrc2:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  34   Thread 0xffff15ffb0b0 (LWP 1214912) "udpsrc3:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  35   Thread 0xffff157fa0b0 (LWP 1214913) "rtpsession-rtcp" 0x0000fffff7e951a4 in __ppoll (fds=0xffff540164a0, nfds=1, timeout=<optimized out>, 
    sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44
  36   Thread 0xffff14ff90b0 (LWP 1214914) "udpsrc5:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  37   Thread 0xfffeff7fe0b0 (LWP 1214916) "timer"           syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  38   Thread 0xfffefffff0b0 (LWP 1214915) "timer"           syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  39   Thread 0xfffefeffd0b0 (LWP 1214917) "rtpjitterbuffer" syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  40   Thread 0xfffefe7fc0b0 (LWP 1214918) "rtpjitterbuffer" syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  41   Thread 0xfffefdffb0b0 (LWP 1214919) "udpsrc6:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  42   Thread 0xfffefd7fa0b0 (LWP 1214920) "timer"           syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  43   Thread 0xfffefcff90b0 (LWP 1214921) "fakesrc3:src"    syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  44   Thread 0xfffedbfff0b0 (LWP 1214922) "rtpjitterbuffer" syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  45   Thread 0xfffedb7fe0b0 (LWP 1214923) "rtpsession-rtcp" __lll_lock_wait (futex=futex@entry=0xffff60009970, private=0) at lowlevellock.c:52
  46   Thread 0xfffedaffd0b0 (LWP 1214924) "udpsrc8:src"     syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  47   Thread 0xfffeda7fc0b0 (LWP 1214925) "udpsrc9:src"     0x0000fffff7e95098 in __GI___poll (fds=0xfffeda7fb268, nfds=2, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  48   Thread 0xfffed9ffb0b0 (LWP 1214926) "timer"           syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  49   Thread 0xfffed97fa0b0 (LWP 1214927) "rtpjitterbuffer" syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  50   Thread 0xfffed8ff90b0 (LWP 1214934) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffeb80c99cc)
    at ../sysdeps/nptl/futex-internal.h:183
  51   Thread 0xfffebffff0b0 (LWP 1214935) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffeb80c3c78)
    at ../sysdeps/nptl/futex-internal.h:183
  52   Thread 0xfffebf7fe0b0 (LWP 1214936) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffeb80f37b8)
    at ../sysdeps/nptl/futex-internal.h:183
  53   Thread 0xfffebeffd0b0 (LWP 1214937) "V4L2_DecThread"  futex_wait_cancelable (private=0, expected=0, futex_word=0xfffeb8941fc8)
    at ../sysdeps/nptl/futex-internal.h:183
  54   Thread 0xfffeb3fff0b0 (LWP 1214938) "stream-muxer:sr" syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
  55   Thread 0xfffeb37fe0b0 (LWP 1214939) "nvv4l2decoder0:" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffeb8941f58)
    at ../sysdeps/nptl/futex-internal.h:183
  56   Thread 0xfffeb2ffd0b0 (LWP 1214940) "python3"         futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, 
    futex_word=0x1b152b0) at ../sysdeps/nptl/futex-internal.h:320
  57   Thread 0xfffeb27fc0b0 (LWP 1214941) "python3"         futex_wait_cancelable (private=0, expected=0, futex_word=0x1e98f90)
    at ../sysdeps/nptl/futex-internal.h:183
  58   Thread 0xfffeb1ffb0b0 (LWP 1214942) "python3"         futex_wait_cancelable (private=0, expected=0, futex_word=0x1e98f94)
    at ../sysdeps/nptl/futex-internal.h:183
  59   Thread 0xfffeb17fa0b0 (LWP 1214943) "python3"         futex_wait_cancelable (private=0, expected=0, futex_word=0x1e98f90)
    at ../sysdeps/nptl/futex-internal.h:183
  60   Thread 0xfffeb0ff90b0 (LWP 1214944) "python3"         futex_wait_cancelable (private=0, expected=0, futex_word=0x1e98f90)
    at ../sysdeps/nptl/futex-internal.h:183
  61   Thread 0xfffe9bfff0b0 (LWP 1214945) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffee80951fc)
    at ../sysdeps/nptl/futex-internal.h:183
  62   Thread 0xfffe9b7fe0b0 (LWP 1214946) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffee808f26c)
    at ../sysdeps/nptl/futex-internal.h:183
  63   Thread 0xfffe9affd0b0 (LWP 1214947) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffee80a92e8)
    at ../sysdeps/nptl/futex-internal.h:183
  65   Thread 0xfffe99ffb0b0 (LWP 1214949) "ctx1080#0"       futex_wait_cancelable (private=0, expected=0, futex_word=0xfffea19e0890)
    at ../sysdeps/nptl/futex-internal.h:183
  66   Thread 0xfffe997fa0b0 (LWP 1214950) "ctx1080#1"       futex_wait_cancelable (private=0, expected=0, futex_word=0xfffea19e0890)
    at ../sysdeps/nptl/futex-internal.h:183
  67   Thread 0xfffe98ff90b0 (LWP 1214951) "ctx1080#2"       futex_wait_cancelable (private=0, expected=0, futex_word=0xfffea19e0890)
    at ../sysdeps/nptl/futex-internal.h:183
  68   Thread 0xfffe91e2a0b0 (LWP 1214952) "ctx1080#3"       futex_wait_cancelable (private=0, expected=0, futex_word=0xfffea19e0890)
    at ../sysdeps/nptl/futex-internal.h:183
  69   Thread 0xfffe916290b0 (LWP 1214953) "ctx1080#4"       futex_wait_cancelable (private=0, expected=0, futex_word=0xfffea19e0890)
    at ../sysdeps/nptl/futex-internal.h:183
  70   Thread 0xfffe90e280b0 (LWP 1214954) "ctx1080#5"       futex_wait_cancelable (private=0, expected=0, futex_word=0xfffea19e0890)
    at ../sysdeps/nptl/futex-internal.h:183
  71   Thread 0xfffe7bfff0b0 (LWP 1214955) "ctx1080#6"       futex_wait_cancelable (private=0, expected=0, futex_word=0xfffea19e0890)
    at ../sysdeps/nptl/futex-internal.h:183
  72   Thread 0xfffe7b7fe0b0 (LWP 1214956) "ctx1080#7"       futex_wait_cancelable (private=0, expected=0, futex_word=0xfffea19e0890)
    at ../sysdeps/nptl/futex-internal.h:183
  73   Thread 0xfffe7affd0b0 (LWP 1214957) "Work queue"      futex_wait_cancelable (private=0, expected=0, futex_word=0xfffea20653f8)
    at ../sysdeps/nptl/futex-internal.h:183
  74   Thread 0xfffe7a7fc0b0 (LWP 1214958) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffed00a9f98)
    at ../sysdeps/nptl/futex-internal.h:183
  75   Thread 0xfffe79ffb0b0 (LWP 1214959) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffed0099adc)
    at ../sysdeps/nptl/futex-internal.h:183
  76   Thread 0xfffe797fa0b0 (LWP 1214960) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffed008d0fc)
    at ../sysdeps/nptl/futex-internal.h:183
  78   Thread 0xfffe5ffff0b0 (LWP 1214962) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffee4009f9c)
    at ../sysdeps/nptl/futex-internal.h:183
  79   Thread 0xfffe5f7fe0b0 (LWP 1214963) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffee408d388)
    at ../sysdeps/nptl/futex-internal.h:183
  80   Thread 0xfffe5effd0b0 (LWP 1214964) "rtpjitterbuffer" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffee40ce8b8)
    at ../sysdeps/nptl/futex-internal.h:183
  81   Thread 0xfffe5e7fc0b0 (LWP 1214965) "cuda-EvtHandlr"  0x0000fffff7e95098 in __GI___poll (fds=0xfffe58000b60, nfds=9, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  82   Thread 0xfffe5dffb0b0 (LWP 1214966) "V4L2_DecThread"  futex_wait_cancelable (private=0, expected=0, futex_word=0xfffee48dd288)
    at ../sysdeps/nptl/futex-internal.h:183
  83   Thread 0xfffe5d7fa0b0 (LWP 1214967) "python3"         futex_wait_cancelable (private=0, expected=0, futex_word=0xffff982ba538)
    at ../sysdeps/nptl/futex-internal.h:183
  85   Thread 0xfffe522050b0 (LWP 1214969) "CPMMListener"    0x0000fffff7e95098 in __GI___poll (fds=0xfffe52204868, nfds=1, 
    timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  88   Thread 0xfffe50a020b0 (LWP 1214972) "nvv4l2decoder3:" futex_wait_cancelable (private=0, expected=0, futex_word=0xfffee48dd21c)
    at ../sysdeps/nptl/futex-internal.h:183
(gdb)
5

You can refer to the link gnu-debugger to learn how to debug ghe hang issue with gdb. I’ll try to use the deepstream_test_3.py to reproduce the issue on my side too.
Could you attach your command so that I can tell if you’re using nvinfer or nvinferserver?

6

This is the command: python3 flavionew.py -o screen -i rtsp://xxx:xxx@10.21.45.19 rtsp://xxx:xxx@10.21.45.19 rtsp://xxx:xxx@10.21.45.19 rtsp://xxx:xxx@10.21.45.19

These are additional facts I collected so far:

  1. nvgstplayer-1.0 is still running for almost 4 days with no problem to get the RTSP stream
  2. The warning gst-resource-error-quark: Could not read from resource. lead me to this post Error with gst-resource-error-quark, but I already changed the batched-push-timeout to fit my 25fps RTSP stream, that is, batched-push-timeout=1000000/25
  3. I changed the camera by another and the freezing problem persists.
  4. I also removed all appliances from lab network, and created a physically segregated network, just in case. Now I have the machine device with the GPU using a static IP, the camera with static IP, and both of them connected to a hub. The freezing still occurs.
  5. At this post Adjusting muxer's batch push timeout based on FPS of fastest source 40000 - #6 by bpheng it is suggested to make streammux.set_property("sync-inputs", False). This property is not described at the manual (Gst-nvstreammux — DeepStream 6.3 Release documentation) but is describer in a new nvstreammux (Gst-nvstreammux New — DeepStream 6.3 Release documentation). Since there was no complain on such property, I suppose I am using this new nvstreammux. However, the freezing persists.
7

I have been running the deepstream_test_3.py all day on my side. There was no “freeze” problem.
As the log attached

This is because the rtsp source in Gstreamer has detected a problem with the server_eof.

So it may related to the Gstreamer open source. Could you upgrade the DeepStream to the latest version? We have upgraded the Gstreamer from 1.16 to 1.20.

8

I can’t upgrade DeepStream, SDK Manager doesn’t allow me to do so.

I also saw this server_eos at the logs, but I doesn’t understand why nvgstplayer is running 24/7 with no problems since May, 11th, when I opened the post. I am using it as a counter-proof to keep an watching to the camera stream.

And when I use Wireshark, I see the stream being interrupted, but its not possible to say if the client stopped consuming the stream, or the server stopped streaming. What I can say is that the “end connection” comes after some seconds (~10s) of idle connection (stream stopped).

9

There may be a reconnection mechanism in gstplayer. Could you try to use nvurisrcbin and set the rtsp-reconnect-interval property to that?
You can refer to Gst-nvurisrcbin specific properties to learn how to set the property.

10

I seems nvurisrcbin doesn’t freeze any more. I set rtsp-reconnect-interval=10s. nvurisrcbin is a wrapper of nvuridecodebin and whatever the former aggregates to the later, it corrects the freezing, and I suspect the reconnecting property isn’t being used since Wireshark doesn’t show any stream interrupt at all. I kept it because it is very useful.

However it now has a side effect, there is a lot of jittering when using 4 sources, and no jittering when using 1 source.

I tried to remove this jittering by setting nvurisrcbin’s latency to 10 and to 1000 (default is 100), but the jittering is still there. Note that in the past I had problems with deepstream_test_3, and I had to set the drop-on-latency=True to remove such jit. But now, using nvurisrcbin, the jittering is back.

11

What is the resolution of your stream? You can try to set the latency to 5000(5s) and the udp-buffer-size to 10485760(10M).

12

The camera is a 1920x1080 resolution with 25fps.

This didn’t reduced the jittering, but introduced a lot o latency (dt between raising my hand and see my hand raised at screen).

By now, my nvurisrcbin properties are like this:

uri_decode_bin = make_elm_or_print_err("nvurisrcbin", "uri-source-bin")
uri_decode_bin.set_property("rtsp-reconnect-interval", 10)
uri_decode_bin.set_property("latency", 5000)
uri_decode_bin.set_property("file-loop", 1)
uri_decode_bin.set_property("cudadec-memtype", 0)
uri_decode_bin.set_property("udp-buffer-size", 10485760)
uri_decode_bin.set_property("uri", uri)
13

OK. Could you try to set the select-rtp-protocol to 4? The issue may be related to the transport layer protocol.

14

This reduced, but not eliminated, the jittering for static objects, and I couldn’t see any difference for moving objetcs (eg: when I raise my hand, or when I move position with my chair) they are still jittering a lot.

I think I understand what you are trying to do, create a pretty good well behavioral stream to be processed by nvurisrcbin. I added drop-frame-interval=2 which had a small improve in jittering, but it is quite visible yet.

Then I tried use nvurisrcbin right from the provider, just like DeepStreeam Plugin Guide with a slight difference since I am using a Jetson (see DeepStream 6 gstreamer pipeline was working in DeepStream 5). Thus, I did:

gst-launch-1.0 nvurisrcbin uri=rtsp://xxxx:xxxx@10.21.45.19 ! m.sink_0 nvstreammux name=m width=1920 height=1080 batch-size=1 ! nvmultistreamtiler ! nvvideoconvert ! nvegltransform ! nveglglessink

gst-launch shows no jittering but it complains a lot about the following. I am not sure if it is related to my problem.

WARNING: from element /GstPipeline:pipeline0/GstEglGlesSink:eglglessink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(3003): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:eglglessink0:
There may be a timestamping problem, or this computer is too slow.

15

From the log and the comment above, it may be that the board is overloaded, or there is a problem with the pts of your stream.

You can try to boost-the-clocks and set the “interval” parameter for nvinfer too.

16

Thank you @yuweiw, now I have a very small jittering for fast moving object, which is pretty much admissible. We made so many changes to reach this and it is quite difficult to summarize all this thread issue. I will assign the “Complete” to the last post, but I strongly recommend further users to read with attention to all the thread. There are many valuable tips.

17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.