Deepstream pipeline after sometime not running pipeline itself stopped

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

**• Hardware Platform -----> GPU
• DeepStream Version -------> 7.0
• TensorRT Version --------> 8.5
• NVIDIA GPU Driver Version (valid for GPU only) ------> GPU 0

While I am running 80 cameras single process it’s running after sometime

(python3:35201): GStreamer-CRITICAL **: 07:41:30.684: gst_buffer_get_size: assertion ‘GST_IS_BUFFER (buffer)’ failed

(python3:35186): GStreamer-CRITICAL **: 07:41:37.757: gst_buffer_get_size: assertion ‘GST_IS_BUFFER (buffer)’ failed

printing and pipeline is stopped ! Not able to understand why this is happening I am using rtsp stream !

@snehashish.debnath

3

Please check CPU/GPU/codec/bandwidth usage. I can’t give any suggestions based on this information alone.

4

The video is having H265

But may i ask what is the co-relation u are looking forward to CPU/GPU/bandwidth usage?

I want to understand why is this “gst_buffer_get_size: assertion ‘GST_IS_BUFFER (buffer)’ failed”
What are the possible cause for this ?

5

80 cameras will take up a lot of system resources. If the CPU/GPU loading is too high, many bugs will appear that will not appear under low loading.

Can you share the steps and sample code to reproduce the problem?

There are quite a few reasons that may cause this error. Please refer to Gstreamer’s code.

gsize
gst_buffer_get_size (GstBuffer * buffer)
{
  guint i;
  gsize size, len;

  g_return_val_if_fail (GST_IS_BUFFER (buffer), 0);

  /* FAST PATH */
  len = GST_BUFFER_MEM_LEN (buffer);
  for (i = 0, size = 0; i < len; i++)
    size += GST_BUFFER_MEM_PTR (buffer, i)->size;
  return size;
}
6

Hi @junshengy

I am using L4 GPU , codec I’m using is H265. I am attaching all the information CPU, GPU and bandwidth
docker stats and nvidia-smi dmon in a .csv format. you can look into and share me some suggestion How to avoid the error.

In this also I am running 20 cameras *4 processes total 80 cameras RTSP stream. After running it sometime two process got down this time 2 processes means 40 cameras is running ! (sometimes one process got down 3 process was running )
l4_all_stats_forum.zip (105.8 KB)

7 
2025-03-12 08:32:05,0,73,55,-,100,56,0,0,0,0,6250,1170
2025-03-12 08:32:07,0,71,55,-,98,61,0,0,0,0,6250,1470
2025-03-12 08:32:08,0,73,56,-,100,54,0,0,0,0,6250,1500
2025-03-12 08:32:09,0,69,56,-,100,55,0,0,0,0,6250,1305

The GPU usage has reached 100%, video memory is almost full. you can refer to the sm/mem columns in l4_nvidia_smi_dmon.csv， You may need to distribute the task to multiple GPUs

2025-03-12 11:08:17,dd0706f2dd90,inspiring_saha,899.91%,15.49GiB / 125.7GiB,12.32%,203GB / 4.59GB,3.33GB / 29.4GB,1157
2025-03-12 11:08:20,dd0706f2dd90,inspiring_saha,888.96%,15.48GiB / 125.7GiB,12.31%,203GB / 4.59GB,3.33GB / 29.5GB,1157
2025-03-12 11:08:23,dd0706f2dd90,inspiring_saha,792.38%,15.48GiB / 125.7GiB,12.31%,203GB / 4.59GB,3.33GB / 29.5GB,1157

How many cores does your CPU have? The usage is also very high

8

Hi @junshengy
I share My code and with you ! Can you please check the code, something wrong in our code please share with us.
We can fix the issue ASAP. This is very argent for us.

9

I don’t understand what your problem is, is it the program exiting abnormally, or a memory leak?

Have you used top/valgrind to check the program’s memory growth?

The decoder usage is not a bottleneck, about 25%-30%，which has nothing to do with H264/H265

In l4_docker_stats.csv, after running the program for 3-4 hours, the memory does not increase significantly.

2025-03-12 08:45:39,dd0706f2dd90,inspiring_saha,811.15%,17.84GiB / 125.7GiB,14.18%,27.9GB / 570MB,3.33GB / 10.9GB,1476
2025-03-12 08:45:42,dd0706f2dd90,inspiring_saha,925.86%,17.77GiB / 125.7GiB,14.13%,28GB / 572MB,3.33GB / 10.9GB,1478
2025-03-12 08:45:45,dd0706f2dd90,inspiring_saha,842.52%,17.84GiB / 125.7GiB,14.18%,28.1GB / 574MB,3.33GB / 10.9GB,1476


2025-03-12 11:08:15,dd0706f2dd90,inspiring_saha,776.11%,15.46GiB / 125.7GiB,12.29%,203GB / 4.59GB,3.33GB / 29.4GB,1157
2025-03-12 11:08:17,dd0706f2dd90,inspiring_saha,899.91%,15.49GiB / 125.7GiB,12.32%,203GB / 4.59GB,3.33GB / 29.4GB,1157
2025-03-12 11:08:20,dd0706f2dd90,inspiring_saha,888.96%,15.48GiB / 125.7GiB,12.31%,203GB / 4.59GB,3.33GB / 29.5GB,1157
2025-03-12 11:08:23,dd0706f2dd90,inspiring_saha,792.38%,15.48GiB / 125.7GiB,12.31%,203GB / 4.59GB,3.33GB / 29.5GB,1157

If the GPU loading is too high and the program is killed by the OS, this is normal, Using the int8 model or increasing the interval property value of nvinfer can reduce the GPU usage.

10

The screenshot does not explain the problem. If there are many detected targets, the memory usage will also increase.

11

@junshengy

Why pipeline getting killed after a period of time. 2 L4 I’m running 160 total cameras 8 processes every GPU, we have 4 pipeline processes. each process we have 20 RTSP stream. Not able to see any error, But killing the pipeline.

12

1.Is it the container that is killed or just a process?
2.Check the kernel and docker service logs

dmesg | grep -i kill

sudo journalctl -u docker.service > docker.log

3.Add faulthandler in python code

13

@junshengy

This error I am getting Can you share us more inside about error to fix the process killed problem.

#0 __pthread_kill_implementation (no_tid=0, signo=11, threadid=140697101403712) at ./nptl/pthread_kill.c:44
tid =
ret = 0
pd = 0x7ff698c00640
old_mask = {__val = {2175, 13488649512622046295, 140737349464432, 89, 93824997943776, 140697101403584, 2, 11, 93824997943776, 93824995642646, 93824995521982, 93824993579512, 0, 47244640256, 18446744073709551615, 0}}
ret =
pd =
old_mask =
ret =
tid =
ret =
resultvar =
resultvar =
__arg3 =
__arg2 =
__arg1 =
_a3 =
_a2 =
_a1 =
__futex =
resultvar =
__arg3 =
__arg2 =
__arg1 =
_a3 =
_a2 =
_a1 =
__futex =
__private =
__oldval =
result =
#1 __pthread_kill_internal (signo=11, threadid=140697101403712) at ./nptl/pthread_kill.c:78
No locals.
#2 __GI___pthread_kill (threadid=140697101403712, signo=signo@entry=11) at ./nptl/pthread_kill.c:89
No locals.
#3 0x00007ffff7c92476 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
ret =
#4
No locals.
#5 0x00007fff8baffebc in gst_buffer_copy_into () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
No symbol table info available.
#6 0x00007fff906fd1d6 in ?? () from /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so
No symbol table info available.
#7 0x00007fff8bb691d7 in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
No symbol table info available.
#8 0x00007fff90142384 in g_thread_pool_thread_proxy (data=) at ../glib/gthreadpool.c:350
task = 0x7ff3dc4a6eb0
pool =
#9 0x00007fff90141ac1 in g_thread_proxy (data=0x7ff9f003e6b0) at ../glib/gthread.c:831
thread = 0x7ff9f003e6b0
func = “g_thread_proxy”
#10 0x00007ffff7ce4ac3 in start_thread (arg=) at ./nptl/pthread_create.c:442
–Type for more, q to quit, c to continue without paging–c
ret =
pd =
out =
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140711796144912, -9073527173854735689, 140697101403712, 0, 140737350879184, 140711796145264, 9077403502112199351, 9073544368178778807}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call =
#11 0x00007ffff7d76850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=11, threadid=140697101403712) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=11, threadid=140697101403712) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140697101403712, signo=signo@entry=11) at ./nptl/pthread_kill.c:89
#3 0x00007ffff7c92476 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
#4
#5 0x00007fff8baffebc in gst_buffer_copy_into () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#6 0x00007fff906fd1d6 in ?? () from /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so
#7 0x00007fff8bb691d7 in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#8 0x00007fff90142384 in g_thread_pool_thread_proxy (data=) at ../glib/gthreadpool.c:350
#9 0x00007fff90141ac1 in g_thread_proxy (data=0x7ff9f003e6b0) at ../glib/gthread.c:831
#10 0x00007ffff7ce4ac3 in start_thread (arg=) at ./nptl/pthread_create.c:442
#11 0x00007ffff7d76850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

14

Please provide kernel log and docker service log to determine why the program was killed. This stack trace cannot determine the problem.

sudo dmesg  > kernel.log

sudo journalctl -u docker.service > docker.log
15

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.