Warnings when streaming videos without b-frames

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hello,

I’ve observed that when I use RTSP streams without B-frames in my DeepStream applications, either by re-encoding the stream to deactivate B-frames or by using the baseline encoding profile, I encounter multiple errors and warnings.

These are the following:

reference in DPB was never decoded
reference in DPB was never decoded
NVDEC_COMMON: NvDecGetSurfPinHandle : Surface not registered 
NVDEC_COMMON: NvDecGetSurfPinHandle : Surface not registered 
NVDEC_COMMON: NvDecGetSurfPinHandle : Surface not registered 
NVDEC_COMMON: NvDecGetSurfPinHandle : Surface not registered 
NVDEC_COMMON: NvDecGetSurfPinHandle : Surface not registered 
NVDEC_COMMON: NvDecGetSurfPinHandle : Surface not registered 
NVDEC_COMMON: NvDecGetSurfPinHandle : Surface not registered 
NVDEC_COMMON: NvDecGetSurfPinHandle : Surface not registered 
NVDEC_COMMON: NvDecGetSurfPinHandle : Surface not registered 
NVDEC_COMMON: NvDecGetSurfPinHandle : Surface not registered

The steps to reproduce are:

  1. Re-encode the sample video to have no b-frames:
cp /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 .
ffmpeg -i sample_720p.mp4 -c:v libx264 -profile:v main -bf 0 -an sample_720p_new.mp4
  1. Stream the new video:
#!/bin/bash

# Start the rtsp-simple-server in the background, the script I use is:
./rtsp-simple-server rtsp-simple-server.yml &

# Give the server a few seconds to start up
sleep 5

ffmpeg -re -stream_loop -1 -i sample_720p_new.mp4 -r 30 -c copy  -f rtsp rtsp://localhost:8554/teststream1 &

# Wait for all background processes to complete
wait
  1. Run the deepstream reference application :
cd /opt/nvidia/deepstream/deepstream-6.3/sources/apps/sample_apps/deepstream-app
sudo ./deepstream-app -c /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/rtsp_test.txt  > ~/log_out.txt 2> ~/log_err.txt

rtsp_test.txt (4.3 KB)
log_out.txt (979 Bytes)
log_err.txt (5.3 KB)

I’ve tried to find a solution for these errors but haven’t been successful. Could you please help me resolve this issue?

Thank you in advance.

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson Orin NX
• DeepStream Version 6.3
• JetPack Version (valid for Jetson only) 5.1.2
• TensorRT Version 8.5.2.2
• NVIDIA GPU Driver Version (valid for GPU only) 11.4.315
• Issue Type( questions, new requirements, bugs) bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) Provided above
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

3

This log should have nothing to do with whether there are b-frames or not.

The following command line can forward the local file to rtsp stream. you can try it.

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 ! h264parse ! nvv4l2decoder ! nvrtspoutsinkbin

This is a BSP issue related to DRM

4

Thanks for your reply.

However, I do not fully understand your response. When It does not concern b-frames, why do the warnings disappear with the normal stream that contains b-frames and only appear once I remove them? Is it dependent on other stream properties?

Furthermore, is there a way to fix the issue on my setup since I can’t change the versions due to contractual reasons and what impact will this issue have on my Deepstream Application?

5

Since I don’t have an Orin NX, I tested your flow on an AGX Orin（DS-7.0), and I used a newer rtsp-simple-server called mediamtx. No similar problems encountered.

Is there any blur/green screen? Try add select-rtp-protocol=4 in [sourcex] group for rtp over tcp

Can you reproduce the problem using the following command?

gst-launch-1.0 uridecodebin uri="rtsp://your rstp camera" ! nvvideoconvert ! video/x-raw"(memory:NVMM)",format=NV12 ! nvv4l2h264enc ! filesink location=out.h264
6

Thank you for your reply.

There is no blur or green screen issue observed.

Regarding your code, I can reproduce the error with the re-encoded stream. Below are the commands I used and the corresponding logs:

  1. Command with the re-encoded sample file streamed as input:
    • gst-launch-1.0 uridecodebin uri="rtsp://10.10.10.94:8554/teststream1" ! nvvideoconvert ! video/x-raw"(memory:NVMM)",format=NV12 ! nvv4l2h264enc ! filesink location=out.h264 > ~/test/log_out.txt 2> ~/test/log_err.txt
    • log_out.txt (725 Bytes)
    • log_err.txt (2.2 KB)
  2. Command with nvidias sample file streamed as input:
  • gst-launch-1.0 uridecodebin uri="rtsp://10.10.10.94:8554/teststream1" ! nvvideoconvert ! video/x-raw"(memory:NVMM)",format=NV12 ! nvv4l2h264enc ! filesink location=out.h264 > ~/test/log_out_nvidia_sample.txt 2> ~/test/log_err_nvidia_sample.txt
  • log_err_nvidia_sample.txt (273 Bytes)
  • log_out_nvidia_sample.txt (760 Bytes)

Please find the log files attached for your reference. Let me know if you need any further information. Furthermore, the issue is also observed with mediamtx, so this can be the reason.

7

This issue seems to be related to the device and Jetpack version.

  1. Try set property num-extra-surfaces of nvv4l2decoder to 18, like nvv4l2decoder num-extra-surfaces=18.
    2.If possible, try upgrading to newer versions of Jetpack and Deepstream.
8

I set num-extra-surfaces to 18 in the reference application’s source configs, but unfortunately, this didn’t resolve the issue. Is there another solution available?

Regarding point 2, as previously mentioned, this isn’t an option due to contractual reasons on my end.

9

Please help dump the stream, I will try to reproduce it in the current version.
Maybe this is a problem with a specific version, but we usually don’t maintain legacy versions

gst-launch-1.0 rtspsrc location="rtsp://xxxx" protocols=4 ! queue ! parsebin ! queue ! h264parse ! mpegtsmux ! filesink location=rtsp.ts
10

Thanks for looking into it. I have dumped the stream as requested and have appended a zipped version of it.
rtsp_ts.zip (36.1 MB)