NvRmChannelSubmit: NvError_IoctlFailed with error code 22

Hi,

in my app I use a pipeline like this one:

nvstreammux ! nvinfer ! nvtracker ! nvvideoconvert ! video/x-raw,format=RGBA ! videoconvert ! video/x-raw,format=RGB ! appsink

nvstreammux is dynamically linked with live sources (rtsp or http).

These source bins are added/deleted dynamically.

I need to convert the frame to RGB for postprocessing-

The app generally works fine but, randomically, after adding/deleting source bins several times, the inference (nvinfer) continues to work but the image conversion fails:

  • reading NvDsFrameMeta I can see the correct detections
  • the image I get on appsink is the same for all the source bins and it doesn’t change (the overlay date/time on the image does not change)

When this error happens the application prints logs such as these ones:

NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)

I can also replicate the same error using the reference app here:

I did some minor modifications to this reference app to use the same pipeline as the one in my app and to cycle forever. Please find attached the modified source code (with .log extension added since I’m not allowed to upload files with c or zip extensions).

I started the modified app this way:

./deepstream-test-rt-src-add-del "rtsp://root:password@192.168.3.172/axis-media/media.amp?resolution=800x600"

after about 100-150 iterations the issue happened, for example:

decodebin child added decodebin226
decodebin child added rtph264depay113
decodebin child added h264parse113
decodebin child added capsfilter113
decodebin child added nvv4l2decoder113
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
decodebin child added decodebin227
decodebin child added rtpmp4gdepay113
decodebin child added aacparse113
decodebin child added avdec_aac113
decodebin new pad audio/x-raw
decodebin new pad video/x-raw
Decodebin linked to pipeline
KLT Tracker Init
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmChannelSubmit: NvError_IoctlFailed with error code 22

so you need to let the app run for a while to replicate the issue.

In my real app the issue happen more frequently than inside the reference app but this probably is due to the fact that my app do more things (and it handles more than 2 streams at the same time) and so the load of the jetson is higher.

I tryed to use nvvidconv instead of nvvideoconvert but nothing changes.

So basically this seems a bug in deepstream sdk or I’m doing something wrong in the same way as the reference app.

I tested the modified reference app on a jetson nano but in my real app I get the same error on both nano and xavier. Can you please confirm and fix or suggest a workaround? Thanks!

deepstream_test_rt_src_add_del.c.log (22.6 KB)

Hi,
Thanks for the test app. We will try to reproduce the issue.

Hi listsw5yds,

We tried to reproduce your issue.
But we always got below errors, are you reproduce the same error before?

Test pipeline:
$ ./deepstream-test-rt-src-add-del file://$DS_SDK_ROOT/samples/streams/sample_1080p_h265.mp4

STATE CHANGE SUCCESS 0x7ed40130d0

Got EOS from stream 1
Calling Start 0
creating uridecodebin for [file:///opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_1080p_h265.mp4]
decodebin child added source
decodebin child added decodebin3
STATE CHANGE ASYNC

decodebin child added qtdemux3
decodebin child added multiqueue3
decodebin child added h265parse3
decodebin child added capsfilter3
decodebin child added aacparse3
decodebin child added avdec_aac3
decodebin child added nvv4l2decoder3
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 279
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 279
decodebin new pad video/x-raw
Decodebin linked to pipeline
decodebin new pad audio/x-raw
KLT Tracker Init
STATE CHANGE SUCCESS

STATE CHANGE SUCCESS 0x7ed40122f0

Calling Stop 0
STATE CHANGE SUCCESS

STATE CHANGE SUCCESS 0x7f100095e0

WARNING from element stream-muxer: No Sources found at the input of muxer. Waiting for sources.
Warning: No Sources found at the input of muxer. Waiting for sources.
Segmentation fault (core dumped)

Hi,

thanks for your test, I can confirm that my modified sample does not work well using a local file as input.

The local file processing ends before the delete_sources callback is called and so the sample exits.

Can you please try to use a network stream? In my test I’m using an rtsp stream from a network camera. This way you should be able to reproduce the reported issue, thanks

Hi,

please test using this public rtsp url:

./deepstream-test-rt-src-add-del "rtspt://root:password@93.63.103.243:443/axis-media/media.amp?resolution=800x600"

you should be able to reproduce the issue after about 100-120 iterations

decodebin child added decodebin232
decodebin child added rtph264depay116
decodebin child added h264parse116
decodebin child added capsfilter116
decodebin child added nvv4l2decoder116
decodebin child added decodebin233
Opening in BLOCKING MODE 
decodebin child added rtpmp4gdepay116
NvMMLiteOpen : Block : BlockType = 261 
decodebin child added aacparse116
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
decodebin child added avdec_aac116
decodebin new pad audio/x-raw
NvMMLiteBlockCreate : Block : BlockType = 261 
decodebin new pad video/x-raw
Decodebin linked to pipeline
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 15, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 15, SyncPointValue = 0)
KLT Tracker Init
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 15, SyncPointValue = 0)
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 15, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 16, SyncPointValue = 0)

the problem does not seem to happen using local files or small files streamed via rtsp, but it happen with live streams from real RTSP camera, thanks!

The reference app leaks ths caps here:

https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/blob/master/runtime_source_add_delete/deepstream_test_rt_src_add_del.c#L142

and executing the provided app against the provided rtsp url there is a file descriptor leak too:

ps aux | grep dee
root      7565 26.9 26.9 13520404 1094908 pts/6 Sl+ 20:03   9:21 ./deepstream-test-rt-src-add-del rtsp://root:password@192.168.3.172/axis-media/media.amp?resolution=800x600

ls -la /proc/7565/fd | wc -l
1022

ls -la /proc/7565/fd | wc -l
1082

ls -la /proc/7565/fd | wc -l
1619

ls -la /proc/7565/fd | wc -l
1789

as you can see the file descriptors increase over the time. The last value is measured just before the app gives the reported error

Hi,
Thanks for providing more information. So it look like it is not specific to

nvvideoconvert ! video/x-raw,format=RGBA

The issue is also present in default reference app?

Hi,

I don’t know if the fd leak is related to the reported issue or if it is a separate issue.

I think this issue cannot be reproduced with the default reference app: it happen only adding/removing bins to nvstreammux without restarting the whole application.

I can run both my application and the refence app for hours with no fd leak and no image conversion issues. The problem happen if I dynamically add/remove bins to nvstreammux.

In my use case I process streams from rtsp cameras, sometime one of the camera can have temporary connection issues and so the source bin exits and I remove it from the muxer. When the camera is available again I add a new source bin for that camera. Another use case is if I want to change the resolution for a camera and so I have to disconnect/connect and so I have to remove/add a source bin to nvstreammux.

Based on the docs here:

my use case should be supported, on page 33 I read

“”"
The muxer supports addition and deletion of sources at run time
“”"

Please find attached a modified deepstream_test_rt_src_add_del.c that fixes the caps leak and doesn’t exit on a bus message error. If you test this app against the provided public rtsp url (or with any rtsp camera) you can easily replicate both the reported issues,

thanks!

deepstream_test_rt_src_add_del.c.log (22.7 KB)

Another info:

If I have some unreachable cameras (no data is sent at all), there is no fd leak and no image conversion issue. I just tested using 2 reachable and 2 unreachable cameras with no issues.

In this case a new bin is added and the deleted and this seems fine. So the problem seems related to add and release pads to/from nvstreammux.

As explained above this is a serious issue if we have to deal with cameras that frequently change the state from reachable to unreachable and vice versa

Hi,
We will reproduce the issue and investigate further. Will update ocne there is further finding.

how about the problem now? I encounter the same problem.

Is there progress now?

Hi,
It is under investigation. Will update once we have new finding.

We are nvidia’s customer. we have encounter a problem. Please help to analyze. I doubt that nvstreammux has a bug. link is “Nvstreammux received signal SIGSEGV when stop rtsp pipeline”. Thank you.

Hi,

Please upgrade to 5.0 GA and try again.

Hi, I’m unable to reproduce using DS5, thank you!