Deepstream 6.2 rtsp output cannot be reformat to HLS

[feicccccccc]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)

• RTX 3090
• RTX 4090

• DeepStream Version

• deepstream:6.2-triton

• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)

• 525.85.12
• 535.54.03

• Issue Type( questions, new requirements, bugs)

• bugs

• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

• see the following article

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

• see the following article

Hi All,

This is the follow up article of https://forums.developer.nvidia.com/t/deepstream-6-2-encoder-abnomral-behaviour/259258

I am using DS to

1. take some rtsp
2. run some model
3. add some visualisation
4. re encode it
5. stream to rtsp server
6. ffmpeg read the rtsp server and change the format to HLS for downstream web browser

I was using DS 6.1.1 (image: deepstream:6.1.1-triton) with Driver version 515.65.01 on RTX 3090 and everything work fine.

Recently I am upgrading my hardware to RTX 4090 since DS 6.2 support ADA. But the output RTSP stream can no longer be process by ffmpeg and change to HLS. RTX 3090 have the same problem with DS 6.2 as well.

To reproduce the result:

1. Start rtsp server:

docker run --rm -it --network=host bluenviron/mediamtx:latest

2. Start ds pipeline inside the container.

gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! nvvideoconvert ! nvv4l2h264enc ! rtspclientsink location=rtsp://localhost:8554/test

3. change format using ffmpeg

ffmpeg -rtsp_transport tcp -i rtsp://localhost:8554/test -c:v copy -f hls test.m3u8

4. test output hls stream

vlc ./test.m3u8

I tried on both RTX 3090, RTX 4090, and multiple driver following the support matrix. DS6.2 always fail, and DS6.1.1 works fine with RTX 3090. DS6.1.1 works fine with RTX 4090 as well, except some core fucntion, for example tracker, does not work using DS 6.1.1.

I change the encoder from nvv4l2h264enc to x264enc in DS 6.2 and everything works fine again.

gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! nvvideoconvert ! x264enc ! rtspclientsink location=rtsp://localhost:8554/test

ffmpeg -rtsp_transport tcp -i rtsp://localhost:8554/test -c:v copy -f hls test.m3u8

From the old post Fiona tried with RTX3090, but it actually didn’t work. ffmpeg did not output the correct segment and list for HLS, which can be observed from the last command line.

ffmpeg -rtsp_transport tcp -i rtsp://localhost:8554/test -c:v copy -f hls test.m3u8
ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[rtsp @ 0x561ddfdb9780] decoding for stream 0 failed
Input #0, rtsp, from 'rtsp://localhost:8554/test':
  Metadata:
    title           : Stream
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
[hls @ 0x561de0161300] Opening 'test0.ts' for writing
Output #0, hls, to 'test.m3u8':
  Metadata:
    title           : Stream
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 30 fps, 30 tbr, 90k tbn, 30 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[hls @ 0x561de0161300] Opening 'test.m3u8.tmp' for writingte=N/A speed=   0x    
**frame=    0 fps=0.0 q=-1.0 Lsize=N/A time=00:00:00.00 bitrate=N/A speed=   0x**

For successful conversion, ffmpeg will show correct frame, fps, etc. And output the segment (.ts) and list (.m3u8) successfully. Check the last command output

ffmpeg -rtsp_transport tcp -i rtsp://localhost:8554/test -c:v copy -f hls test.m3u8
ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, rtsp, from 'rtsp://localhost:8554/test':
  Metadata:
    title           : Stream
  Duration: N/A, start: 1.593656, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
[hls @ 0x555df3f3ae80] Opening 'test0.ts' for writing
Output #0, hls, to 'test.m3u8':
  Metadata:
    title           : Stream
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 30 fps, 30 tbr, 90k tbn, 30 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[hls @ 0x555df3f3ae80] Opening 'test.m3u8.tmp' for writingte=N/A speed=1.06x    
**frame=  224 fps= 18 q=-1.0 Lsize=N/A time=00:00:13.06 bitrate=N/A speed=1.06x**

Since the it works by just changing from hardware deocoding to software decoding. I suspect the issue is related to the driver and encoder. Please kindly advice and it is urgent to get the RTX 4090 working with deepstream. Many Thanks!

[feicccccccc]

Update:

It works by using h265 encoder, and ffmpeg is able to parse it to HLS format. The bad news is browser can only support h264 HLS format.

i.e.

gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! nvvideoconvert ! nvv4l2h265enc ! rtspclientsink location=rtsp://localhost:8554/test

[feicccccccc]

Another observation is:

The stream show compression artifact using VLC player when using nvv4l2h264enc. It does not happen using nvv4l2h265enc.

[413405663]

I also had a similar issue.

I stream my output to ZLMediaKit server with rtmp (... ! nvv4l2h264enc ! flvmux streamable=true ! rtmpsink location="rtmp://..."), and the resulting stream shows lots of gray/green blocks constantly during playback in VLC, similar to your screenshot. And stream transcoding/transmuxing to HLS/HTTP-FLV/WebRTC doesn’t work, as VLC and browser clients stuck at “starting playback” and refuse to show any video frames.

Changing encoder to x264enc and everything works fine, but HtoD/DtoH data transfer will introduce negative performance impact.

Changing encoder to nvh264enc also works, but since I can’t find a way to convert video/x-raw(memory:NVMM) buffers to video/x-raw(memory:GLMemory) or video/x-raw(memory:CUDAMemory), I must first transfer the data to host memory, then back to device memory again (... ! nvvideoconvert ! video/x-raw ! nvh264enc ! ...), which is also a huge performance hit.

Then I found a fix to this issue when fiddling with nvv4l2h264enc’s parameters. Set tuning-info-id to HighQualityPreset, while decoding error gray/green artifacts still exist, they eventually disappear after a few seconds. And HLS/HTTP-FLV/WebRTC transcoding/transmuxing works without issue.

Maybe there is something wrong with the default LowLatencyPreset.

For reference, I am using Deepstream 6.2 (deepstream:6.2-devel) with RTX3060.

[feicccccccc]

Thanks! 413405663. Changing the preset solve the problem. To be specific, using LowLatencyPreset , and UltraLowLatencyPreset seems create pts problem.

[yuweiw]

Hi @feicccccccc , could you post your parameter settings in detail? It can be used as a reference for those who encounter similar problems in the future. Thanks

[feicccccccc]

sure. let me post the launch string here.

Card: RTX4090
Driver: 525.85.12
Image: deepstream:6.2-triton

launch string:

gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! nvvideoconvert ! nvv4l2h264enc tuning-info-id=1 ! rtspclientsink location=rtsp://localhost:8554/test

[feicccccccc]

Yes I was performing test on individual parameters. Let me briefly descirbe my finding.

• Same Deepstream Version with different cards.

[yuweiw]

You changed too many variables at once during your testing. Could you control only one variable at a time for the test? Like:

• Same Deepstream Version with different cards.
• Different DeepStream Versions with same card.
• Different encoder with the same DeepStream version and the same card

And what is the probability of this problem occurring？

[feicccccccc]

Yes sure. I was performing tests on individual parameters and let me briefly describe my finding.

• Using nvv4l2h264enc

	3090	4090
DS 6.1.1	No significant error	No significant error
DS 6.2	encoder problem	encoder problem

• Using nvv4l2h265enc

	3090	4090
DS 6.1.1	No significant error	No significant error
DS 6.2	encoder problem	encoder problem

• Using x264enc

	3090	4090
DS 6.1.1	No significant error	No significant error
DS 6.2	No significant error	No significant error

On top of that DS 6.1.1 does not support 4090, and nvtracker is not available and encounters Cuda problem when launch.

Please let me know if you have any questions. Been trying to make 4090 work since it was meant to be a hardware upgrade for the DS applications. Many Thanks!

[yuweiw]

@feicccccccc

On top of that DS 6.1.1 does not support 4090, and nvtracker is not available and encounters Cuda problem when launch.

This may be a compatibility issue.
I have tried this with DeepStream 6.2 docker on T4. It has no artifact in the video.
Some of the software in our Docker relies on hosts，like cuda, driver…
So if you are using different DeepStream versions in Docker, it is better to have corresponding cuda, driver versions on your host as well. dGPU model Platform and OS Compatibility

[feicccccccc]

Yes I do match the host driver and cuda based on the compatibility matrix.
What I meant is I can start a raw launch string using DS6.1.1 with 4090, but I cannot use other plugins, like nvtracker, since 4090 is not supported by DS 6.1.1 (as shown in the compatibility matrix).

[yuweiw]

@feicccccccc

What I meant is I can start a raw launch string using DS6.1.1 with 4090, but I cannot use other plugins, like nvtracker, since 4090 is not supported by DS 6.1.1 (as shown in the compatibility matrix).

About this problem, you can open a new topic.
In theory, if your driver, cuda, and other sofware versions are based on the compatibility matrix, the DeepStream can be supported.

About the original question, we have tried the pipeline in our environment both on 3090 and 4090. It works well. You can also help confirm the following scenario：use vlc to play the rtsp source directly and see if there is artifact problem.

[feicccccccc]

It actually happen every time I tried with the sample mp4, as well as rtsp stream.

Taken on 4090, with the above mentioned method.

[yuweiw]

It’s weird cause we cannot reproduce the issue in our 4090 or 3090 card. Could you try to use the pipeline to save the video directly without rtspsink?

gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=test.mp4

[feicccccccc]

To make it simpler for us to debug, let me summarise my method, results, as well as screen capture for reference.

1st Experiment:

• Use filesink to decode and reencode the stream to mp4

Setup:

1. DS 6.2
2. DS 6.1.1

Methods:

1. Run the follow gst launch string

gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=test.mp4

2. observe the output mp4

1st Experiment Result:

1. DS 6.2
   • no significant problem
2. DS 6.1.1
   • no significant problem

Output Video: test_mp4.zip (45.5 MB)

2nd Experiment:

• live streaming using rtsp, and reformat to hls using ffmpeg

Setup:

1. DS 6.2

Methods:

1. create a continuous rtsp stream from the testing mp4, and send it to a rtsp server

ffmpeg -re -fflags +genpts -stream_loop -1 -i /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 -an -c:v copy -f rtsp rtsp://localhost:8554/raw

2. decode, and reencode the stream using deepstream
   option:
   • nvv4l2h264enc
   • nvv4l2h264enc tuning-info-id=1
   • nvv4l2h265enc
   • x264enc

gst-launch-1.0 uridecodebin uri=rtsp://192.168.51.100:8554/raw ! nvvideoconvert ! (different encoding option) ! rtspclientsink location=rtsp://localhost:8554/test

3. play with VLC and observe
4. use ffmpeg to transform the rtsp to hls

ffmpeg -use_wallclock_as_timestamps 1 -rtsp_transport tcp -i rtsp://localhost:8554/test -c:v copy -f hls -hls_list_size 3 -hls_time 2 test.m3u8

2nd experiment result

1. nvv4l2h264enc

   • Does not work with ffmpeg reformat to hls
   • obvious compression defect
     DS62_nv264.zip (19.4 MB)

2. nvv4l2h264enc tuning-info-id=1

   • Work with ffmpeg reformat to hls
   • obvious compression defect
   • incorrect segment length
   • takes time to “warm up” before ffmpeg able to reformat
     DS62_nv264_id1.zip (26.7 MB)

3. nvv4l2h265enc

   • works perfectly
   • correct segment length
     DS62_nv265.zip (32.2 MB)

4. x264enc

   • works
   • incorrect segment length
     DS62_x264.zip (40.2 MB)

3rd Experiment

• add videorate to limit the fps

Setup:

1. DS 6.2

Methods:
Same as experiment 2, but adding videorate to the pipeline

Why:
Lower the computation for better performance, but

• more compression defect
• incorrect segment length, which could up to 1 min

3rd experiment result

1. nvv4l2h264enc

   • more compression defect
   • fail to reformat to hls
     DS62_nv264_fps8.zip (30.9 MB)

2. nvv4l2h264enc tuning-info-id=1

   • compression defect
   • can reformat to hls, but takes a long time to “warm up”
   • incorrect segment length
     DS62_nv264_id1_fps8.zip (56.6 MB)

3. nvv4l2h265enc

   • Can’t play through vlc
   • can reformat to hls with correct segment length
     DS62_nv265_fps8.zip (26.8 MB)

4. x264enc

   • ok, slighly incorrect segment legnth
     DS62_x264_fps8.zip (34.8 MB)

Conclusion

using h265 encoder is the best scenario, but our application only supports h264 for web streaming.

while using tuning-info-id=1 for h264 encoder is fine, it does not work well with hls, which generate incorrect and large segment length, and is against the purpose of streaming.

It’s a large amount of files and results. Please let me know if there is anything unclear, and I can explain more.

[feicccccccc]

Is it possible to use the old DS 6.1.1 decoder/encoder/nvvideo4linux for DS 6.2 as a temporary solution? It’s quite urgent and it seems easier.

[yuweiw]

No. We do not support this operation.
About these three rounds of testing, do they both run on 4090 and DS6.1.1 works perfectly for all the scenario?
I have tried the methods below to generate the rtsp source. When playing directly with VLC, large number of artifacts will apear too.

ffmpeg -re -fflags +genpts -stream_loop -1 -i /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 -an -c:v copy -f rtsp rtsp://localhost:8554/raw

[feicccccccc]

Yes, It works perfectly with DS 6.1.1. (except other deepstream components, for instance, nvtracker, does not work on DS6.1.1 with 4090)
May I ask how long it takes to solve the problem? I am planning to hry new machines with 4090, but it seems it will be easier at the moment to get 3090 instead.

[yuweiw]

The picture I posted does not mean that the problem can be reproduced on my end. This is the effect of using the ffmpeg command you provided to generate an rtsp stream, and then playing it directly in VLC. This proves that the rtsp source has problems.
About the issue of nvtracker not working on 4090, you can open a new topic.
We are trying to reproduce the problem on our end.