Deepstream pipeline crashes after period of time due to nvv4l2decoder error

**• Hardware Platform GPU - 2 x A2
**• DeepStream Version 6.3
**• TensorRT Version 8.5.3.1
**• NVIDIA GPU Driver Version - 535.129.03
**• Issue Type - bug

I have a pipeline that functions as expected for anywhere from 30-120 minutes before crashing.

The whole pipeline takes N number of streams via RTSP, performs inference, and then splits into branches that:

  • periodically save videos
  • restream each stream with osd using RTSP
  • restream a lower resolution copy of each stream using RTSP
  • combine the streams using a tiler and stream that over RTSP

This pipeline was made using the Python SDK and most of the setup of components was copied closely from the sample applications.

The only errors and warnings I am seeing when the pipeline crashes are:

1:39:03.279810912     1 0x7e7f700586a0 WARN            v4l2videodec gstv4l2videodec.c:2305:gst_v4l2_video_dec_decide_allocation:<nvv4l2decoder104> Duration invalid, not setting latency
1:39:03.280148349     1 0x7e7f700586a0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1114:gst_v4l2_buffer_pool_start:<nvv4l2decoder104:pool:src> Uncertain or not enough buffers, enabling copy threshold
1:39:03.281162424     1 0x7e7f480166a0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1565:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder104:pool:src> Driver should never set v4l2_buffer.field to ANY
1:39:03.295882329     1      0x2fd3b00 ERROR          v4l2allocator gstv4l2allocator.c:1390:gst_v4l2_allocator_qbuf:<nvv4l2decoder103:pool:src:allocator> failed queueing buffer 3: Bad file descriptor
1:39:03.295905396     1      0x2fd3b00 ERROR         v4l2bufferpool gstv4l2bufferpool.c:1456:gst_v4l2_buffer_pool_qbuf:<nvv4l2decoder103:pool:src> could not queue a buffer 3
1:39:03.295939111     1      0x2fd3b00 WARN           v4l2allocator gstv4l2allocator.c:838:gst_v4l2_allocator_stop:<nvv4l2decoder103:pool:src:allocator> error releasing buffers 

In the case above these appeared ~90 seconds before the pipeline crashed. These errors can also appear multiple times before it crashes so may not be relevant.

I have been unable to debug this issue and was wondering if anyone can provide any advice.

Can you describe “the pipeline crashed”? Is there segmentation fault? Is there any other abnormal things you obeservd when crashing?

Hi Fiona. Would a seg fault show up in the logs?
I am running Deepstream inside a docker container using
FROM nvcr.io/nvidia/deepstream:6.3-triton-multiarch
And apt installing all the dependencies for gst-python. With GST_DEBUG set to 3 there is nothing unusual in the logs at the time of the crash, the docker container just stops.

@michaells Seg fault is reported by OS, you can see it in the log.

Do you mean the whole container freezed? Or just the app stopped without any more log output?

I don’t believe the container freezes it just stops as Deepstream process has stopped.

So it is the app stopped and exit, right?

Have you tried to run the app in the host? You may get more information when there any exit

Yes, stops and exits with no Errors. I haven’t tried running it as host but I’ll see if I can get that working.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

From your description and log, we do not get any clue for what caused the exit.

Please try to find more useful clue for the issue. It is better to monitor the GPU and CPU loading and resource occupancy while running your app too.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.