Producing H265 video in fMP4 and HLS container using nvv4l2h265enc, mp4mux and hlssink2

kang12 · July 27, 2024, 8:42pm

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): both dGPU & Jetson
• DeepStream Version: Deepstream6.4 (dGPU) and Deepstream6.2 (Jetson)
• JetPack Version (valid for Jetson only): Jetpack 5.1.2
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description): Gstreamer version 1.20.3

Hi.
I’m developing a python-based deepstream pipeline to generate H265 encoded video with HLS container.
When I use the following code, it works well and the video is well generated.

        original_width, original_height = py_.at(self.video_meta.resolution, 'width', 'height')
        self.video_appsrc = Gst.ElementFactory.make("appsrc", "video_src")
        self.video_appsrc.set_property("format", Gst.Format.TIME)
        self.video_appsrc.set_property("is-live", True)
        self.video_appsrc.set_property("max-bytes", 0)
        video_src_caps = Gst.caps_from_string(f"video/x-raw(memory:NVMM),format=NV12,width={original_width},height={original_height},framerate={self.fps}/1")
        self.video_appsrc.set_property("caps", video_src_caps)
        self.pipeline.add(self.video_appsrc)

        pre_encoder_queue = Gst.ElementFactory.make("queue")
        pre_encoder_queue.set_property('max-size-buffers', 0)
        pre_encoder_queue.set_property('max-size-bytes', 0)
        pre_encoder_queue.set_property('max-size-time', 0)
        self.pipeline.add(pre_encoder_queue)

        encoder = Gst.ElementFactory.make("nvv4l2h265enc")
        encoder.set_property("control-rate", 1) # constant bitrate
        encoder.set_property('force-idr', True)
        encoder.set_property("idrinterval", self.fps * self.segment_length)
        encoder.set_property('force-intra', True)
        encoder.set_property("iframeinterval", self.fps * self.segment_length)
        encoder.set_property("bitrate", self.bit_rate)
        encoder.set_property('vbvbufsize', self.bit_rate) # https://forums.developer.nvidia.com/t/random-blockiness-in-the-picture-rtsp-server-client-jetson-tx2/174896/5?u=kang12
        encoder.set_property("preset-id", 2) # FastPreset
        encoder.set_property("maxbitrate", int(self.bit_rate * self.max_bitrate_factor))
        encoder.set_property("tuning-info-id", 1)
        self.pipeline.add(encoder)

        capsfilter = Gst.ElementFactory.make("capsfilter", "encoder_caps")
        encoder_caps_string = Gst.Caps.from_string(f"video/x-h265, stream-format=(string)byte-stream,framerate={self.fps}/1")
        capsfilter.set_property("caps", encoder_caps_string)
        self.pipeline.add(capsfilter)

        encoder_parse = Gst.ElementFactory.make("h265parse") # necessary to create hls ts files properly
        encoder_parse.set_property('config-interval', -1)  # Send SPS and PPS with every IDR frame
        self.pipeline.add(encoder_parse)

        self.mux = Gst.ElementFactory.make("hlssink", 'hlssink')
        self.mux.set_property("location", f"{self.local_dir}/{self.segment_prefix}%05d.ts")
        self.mux.set_property("playlist-location", f"{self.local_dir}/{self.segment_prefix}-temp.m3u8")
        self.mux.set_property("target-duration", self.segment_length)
        self.mux.set_property("max-files", 0)
        self.mux.set_property("playlist-length", 0)
        self.pipeline.add(self.mux)

        self.video_appsrc.link(pre_encoder_queue)
        pre_encoder_queue.link(encoder)
        encoder.link(capsfilter)
        capsfilter.link(encoder_parse)
        encoder_parse.link(self.mux)

# I also have an audio stream and link it properly to the hlssink2's audio pad

However, the issue is that when I produce .ts segment based H265 HLS stream, then it’s not playable in Chrome or Safari browsers. I should use fragmented MP4 to make it playable in those browsers.

So I added this in between the h265parse and hlssink2 element in the following way

        self.mp4mux = Gst.ElementFactory.make("mp4mux")
        self.mp4mux.set_property("fragment-duration", self.segment_length * 1000) # ms
        self.pipeline.add(self.mp4mux)
#[...]
        encoder_parse.link(self.mp4mux)
        self.mp4mux.link(self.mux)

and changed a property of the hlssink element in this way to generate .m4s file instead of .ts file

        self.mux.set_property("location", f"{self.local_dir}/{self.segment_prefix}%05d.m4s")

So this is the full code including mp4mux.

        original_width, original_height = py_.at(self.video_meta.resolution, 'width', 'height')
        self.video_appsrc = Gst.ElementFactory.make("appsrc", "video_src")
        self.video_appsrc.set_property("format", Gst.Format.TIME)
        self.video_appsrc.set_property("is-live", True)
        self.video_appsrc.set_property("max-bytes", 0)
        video_src_caps = Gst.caps_from_string(f"video/x-raw(memory:NVMM),format=NV12,width={original_width},height={original_height},framerate={self.fps}/1")
        self.video_appsrc.set_property("caps", video_src_caps)
        self.pipeline.add(self.video_appsrc)

        pre_encoder_queue = Gst.ElementFactory.make("queue")
        pre_encoder_queue.set_property('max-size-buffers', 0)
        pre_encoder_queue.set_property('max-size-bytes', 0)
        pre_encoder_queue.set_property('max-size-time', 0)
        self.pipeline.add(pre_encoder_queue)

        encoder = Gst.ElementFactory.make("nvv4l2h265enc")
        encoder.set_property("control-rate", 1) # constant bitrate
        encoder.set_property("profile", 2) # Main
        encoder.set_property("idrinterval", self.fps * self.segment_length)
        encoder.set_property("iframeinterval", self.fps * self.segment_length)
        encoder.set_property("bitrate", self.bit_rate)
        encoder.set_property('vbvbufsize', self.bit_rate) # https://forums.developer.nvidia.com/t/random-blockiness-in-the-picture-rtsp-server-client-jetson-tx2/174896/5?u=kang12
        encoder.set_property("preset-id", 2) # FastPreset
        encoder.set_property("maxbitrate", int(self.bit_rate * self.max_bitrate_factor))
        encoder.set_property("tuning-info-id", 1)
        self.pipeline.add(encoder)

        capsfilter = Gst.ElementFactory.make("capsfilter", "encoder_caps")
        encoder_caps_string = Gst.Caps.from_string(f"video/x-h265, stream-format=(string)byte-stream,framerate={self.fps}/1")
        capsfilter.set_property("caps", encoder_caps_string)
        self.pipeline.add(capsfilter)

        encoder_parse = Gst.ElementFactory.make("h265parse") # necessary to create hls ts files properly
        self.pipeline.add(encoder_parse)

        self.mp4mux = Gst.ElementFactory.make("mp4mux")
        self.mp4mux.set_property("fragment-duration", self.segment_length * 1000) # ms
        self.pipeline.add(self.mp4mux)

        self.mux = Gst.ElementFactory.make("hlssink", 'hlssink')
        self.mux.set_property("send-keyframe-requests", False)
        self.mux.set_property("location", f"{self.local_dir}/{self.segment_prefix}%05d.m4s")
        self.mux.set_property("playlist-location", f"{self.local_dir}/{self.segment_prefix}-temp.m3u8")
        self.mux.set_property("target-duration", self.segment_length)
        self.mux.set_property("max-files", 0)
        self.mux.set_property("playlist-length", 0)
        self.pipeline.add(self.mux)

        self.video_appsrc.link(pre_encoder_queue)
        pre_encoder_queue.link(encoder)
        encoder.link(capsfilter)
        capsfilter.link(encoder_parse)
        encoder_parse.link(self.mp4mux)
        self.mp4mux.link(self.mux)

But when I add this mp4mux element, the pipeline running, but the following errors are displayed and the video files are corrupted.

(python3:1065): GStreamer-CRITICAL **: 22:20:25.713: gst_segment_to_running_time: assertion 'segment->format == format' failed
0:00:34.316468895  1065 0x5cfd5451b640 INFO        GST_ELEMENT_PADS gstelement.c:1016:gst_element_get_static_pad: found pad hlssink:sink
0:00:34.316521984  1065 0x5cfd5451b640 INFO                 hlssink gsthlssink.c:510:schedule_next_key_unit:<hlssink> sending upstream force-key-unit, index 1 now 99:99:99.999999999 target 0:00:14.999999999
0:00:34.316576682  1065 0x5cfd5451b640 INFO               h265parse gsth265parse.c:3365:gst_h265_parse_src_event:<h265parse0> received upstream force-key-unit event, seqnum 2336 running_time 0:00:14.999999999 all_headers 1 count 1
0:00:34.316636099  1065 0x5cfd5451b640 ERROR                hlssink gsthlssink.c:518:schedule_next_key_unit:<hlssink> Failed to push upstream force key unit event

Is there anything else I should amend the element options or configurations?

I’ve spent a lot of time debugging it but couldn’t make it done and become very desperate.
Really appreciate any helps in advance.

fanzh · July 29, 2024, 2:43am

about “then it’s not playable in Chrome or Safari browsers”, can the ts be played by ffplay or VLC or other players?
about “I should use fragmented MP4 to make it playable in those browsers”, do you mean the recorded mp4 files can be played by those browsers?
from the error “segment->format == format”, there is gstreamer negotiation error. you can modify the following simplified gst-luanch pipeline to debug first. then port to the Python code.

gst-launch-1.0 videotestsrc is-live=true ! x264enc ! h264parse ! hlssink2 max-files=5

kang12 · July 29, 2024, 6:48am

Hi @fanzh Thanks for your comment.

about “then it’s not playable in Chrome or Safari browsers”, can the ts be played by ffplay or VLC or other players?
→ Yes .ts segmented HLS stream is playable through VLC. I found some discussions in different sources that this approach is not supported yet in most of the browsers so the only way is to use fMP4 + HLS which was the reason I tried the new way that we are discussing.
about “I should use fragmented MP4 to make it playable in those browsers”, do you mean the recorded mp4 files can be played by those browsers?
→ The same comment as above. I’ve been using .ts segmented HLS stream so far. There has been no issue when it’s H264 encoded. However, with H265 it’s not playable in the browsers.
I found this post and would like to obtain exactly the same output as this post did with ffmpeg.
HLS and Fragmented MP4 | HTTP Live Streaming
from the error “segment->format == format”, there is gstreamer negotiation error. you can modify the following simplified gst-luanch pipeline to debug first. then port to the Python code.
→ Thanks for your advice. Let me try this and get back to you.

But could you give me some advice whether

Is it common and normal to use fMP4 and HLS for H265 encoded videos?
Is there any other container except for HLS to play H265 video smoother? The bottom line is that the video should be streamable in all browsers.

fanzh · July 30, 2024, 7:54am

you can use use rtsp protocol, please refer to ready-made rtsp output sample deepstream_test1_rtsp_out.py.

kang12 · July 30, 2024, 8:13am

Hi @fanzh appreciate your answer.
However, I’m already aware of the rtsp output and using it for different purposes.

I need to create a file based stream protocol like HLS to make it seekable by users in real-time.
As far as I know rtsp output cannot be seekable unless we save it as a file again additionally on the client side.

I know fMP4 + HLS concept is working through ffmpeg but I couldn’t found any reference in Gstreamer, nor make it work yet.

Any advice to make this H265 based file streaming works, that’d be much appreciated.

fanzh · July 31, 2024, 8:18am

Thanks for the sharing! this issue would outside of DeepStream. there is no ready-made GStreamer sample currently.

yingliu · August 27, 2024, 6:15am

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

system · September 10, 2024, 6:15am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
using gstreamer to transcode/downscale 1080i videos on Jetson TX2 Jetson TX2	19	3856	October 18, 2021
How to run H265 RTSP IP camera in deepstream_app DeepStream SDK	8	4715	August 17, 2021
DS 6.2 6.3 h264 encoder compression artifact and HLS parsing problem DeepStream SDK	16	1039	September 1, 2023
Hardware-accelerated video encoding with gstreamer Jetson TX1	9	11833	October 18, 2021
Synchronize output of different pipelines DeepStream SDK gstreamer	12	45	August 2, 2024
How to transcode H264 rtp packet to H265 rtp packet Jetson AGX Xavier gstreamer	16	3318	January 5, 2022
DeepStream can't decode certain H265 streams properly DeepStream SDK nvbugs	49	2480	November 1, 2024
How to create an rtsp sink with deepstream python program? DeepStream SDK	28	8652	October 12, 2021
Failed to stream GPU encoded rtsp-server using gstreamer DeepStream SDK rtsp , cuda , gstreamer , encoder	12	142	November 28, 2024
h265 encoding only is Level4.1 Jetson TX1	5	1623	October 18, 2021

Producing H265 video in fMP4 and HLS container using nvv4l2h265enc, mp4mux and hlssink2

Related topics