Hardware decoder / resize / encoder pipeline randomly crashes

I have been struggling this issue quite a lot and tried almost everything I can think of or find but couldn’t find any clue.

I’m on Nvidia Jetson Xavier NX with Jetpack 35.4.1 version. I’m using Deepstream6.3 docker image.
I have a python application with some C++ custom elements which looking like this:
rtspsrc → rtpjitterbuffer → rtph265depay → h265parse → queue → nvv4l2decoder → videorate → capsfilter → nvvideoconvert (for resizing) → capsfilter → appsink as one thread
And another thread:
appsrc → nvvideoconvert (for ensuring memory) → nvv4l2h264enc → capsfilter → hlssink2
And the other thread:
appsrc → nvvideoconvert (for ensuring memory) → queue → nvdsvideotemplate (for my custom element)

I know you need more logs but I’d really desperate to have more logs too.
The symptom is that the whole docker process is just crashed without leaving any logs. I added as many logs as possible but when it’s crashed there’s literally nothing which makes the debugging so difficult.

The only small clue is that I found this log in syslog but doesn’t seem to be a direct cause.

nvmap_alloc_handle: PID 936878: python3: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

Could someone help me at least to leave any logs for further debuggings or any insights where I should start to look iinto?
We are using Jetson Xavier NX for our production application and this happens only at some places occasionally but when it happens, it damages our services a lot.
I’d like to fully find out the root cause so we can take any actions accordingly.

I found this conversation is somewhat similar but anyway this user also seems to suffer the similar challenges as I do, which is no log situation.

Thanks.

Hi,
We would suggest break down the pipelines to get more information. Please try only the two threads:

rtspsrc → rtpjitterbuffer → rtph265depay → h265parse → queue → nvv4l2decoder → videorate → capsfilter → nvvideoconvert (for resizing) → capsfilter → appsink as one thread

appsrc → nvvideoconvert (for ensuring memory) → nvv4l2h264enc → capsfilter → hlssink2

And see if random crash occurs.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.