Dual-GPU Deepstream VideoRate Issue

• Hardware Platform (Jetson / GPU)
2 x NVIDIA GeForce RTX 3080

• DeepStream Version
nvcr.io/nvidia/deepstream:6.1-devel

• NVIDIA GPU Driver Version (valid for GPU only)
NVIDIA-SMI 510.73.05 Driver Version: 510.73.05 CUDA Version: 11.6

• Issue Type( questions, new requirements, bugs)

I have found a Deepstream defect where the use of a videorate filter after a uridecodebin3 element in a multi-gpu environment throws the exception below. This is relevant for this forum as the uridecodebin3 element uses the nvv4l2decoder and stores the output buffer in the GPU memory.


GPUassert_VPI: VPI_ERROR_INVALID_OPERATION Container doesn't have any of the necessary backends enabled src/modules/cuDCFv2/featureExtractor.cu 527

GPUassert: invalid device function src/modules/cuDCFv2/cuDCFFrameTransformTexture.cu 694

If running on a single gpu the videorate element after a uridecodebin3 element behaves correctly with Deepstream.

The following command can run on our server which has 4 GPUs.

gst-launch-1.0 uridecodebin3 uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h265.mp4 ! nvvideoconvert ! videorate ! 'video/x-raw,framerate=20/1' ! fakesink

Can you tell us how to reproduce your failure?

Hi,
I will have to write a python minimal example. From what I can tell forcing the gpu-id on the nvv4l2decoder created by the uridecodebin3 (via the ‘pad-added’ callback) interferes with the videorate and downstream inference engines when running with multiple GPUs.

I encountered exactly the same error on mutiple T4 card environment, DS 6.1 and CUDA Driver 510.

I’m trying running two pipelines with same config and different inputs, and I carefully set elements in these two pipelines to use different GPUs, then the error occours.

Interestingly, when I set the two pipelines to use the same GPU, it works with no error!

Hi @zhy2022

We have moved away from videorate to a custom one and so i can see if there are still issues.

Deepstream being closed-source is a mistake… it is just so difficult to debug anything.

I have just tested another config, and this error disappeared.

The difference is nvtracker plugin’s config:
ll-lib-file keeps to/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so, but
ll-config-file changed from an config_tracker_NvDCF_xxx.yml modification to config_tracker_IOU.yml.

I can almost determine the error source: the NvDCF impl in libnvds_nvmultiobjecttracker.

btw, I don’t use videorate to adjust frame rate, but used the drop-frame-interval of plugin nvv4l2decoder.

1 Like

Issue metioned on this page shouble be the same. It means the problem has been there for a while.

1 Like

Excellent debugging both of you.

I should have read the message more carefully which does indicate NvDCF. I found that this issue would take a while to eventuate (it was not easy to produce a deterministic test) so I can test more with both the trackers to try to produce a deeper confirmation.

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Hi @zhy2022 Do you have a way to reproduce this issue? Thanks to share with us if you have.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.