How to create MP4 video playable on web browser?

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

The issue had been there perhaps for a while but only got caught until recently as the project moves on to user interface. We found the MP4 videos generated by GStreamer were not playable on browser. We tried to play the file on IE, Edge, Firefox, Chrome, none of them took the MP4. Well, to be precise, only if VLC was not embedded.

In other words, the MP4 files worked fine on VLC, either in local file mode or in network stream (http) mode. That’s part of the reason we didn’t realized there could be a problem. Given MP4 is such a widely used format, I was a little surprised by running into the road block.

For the purpose of troubleshooting, I created a webpage with a MP4 I created on TX2 and a short video from NASA for reference, and in each case I applied two methods of access, by href link and by HMTL5 video tag.
http://jmrcubed.com/cdn_v07/sdb/demo/mp4av/

Given the URL addresses, I ran ffprobe to compare the stream formats, using the command
ffprobe -i -show_format -hide_banner -show_streams

For the NASA video (playable on all major browsers and VLC), I got:

[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=High
codec_type=video
codec_time_base=1/60
codec_tag_string=avc1
codec_tag=0x31637661
width=1280
height=720
coded_width=1280
coded_height=720
has_b_frames=0
sample_aspect_ratio=1:1
display_aspect_ratio=16:9
pix_fmt=yuv420p
level=31
color_range=N/A
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=left
timecode=N/A
refs=1
is_avc=1
nal_length_size=4
id=N/A
r_frame_rate=30/1
avg_frame_rate=30/1
time_base=1/30
start_pts=0
start_time=0.000000
duration_ts=3493
duration=116.433333
bit_rate=2527807
max_bit_rate=N/A
bits_per_raw_sample=8
nb_frames=3493
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=1
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
TAG:creation_time=1970-01-01 00:00:00
TAG:language=und
TAG:handler_name=VideoHandler
[/STREAM]
[STREAM]
index=1
codec_name=aac
codec_long_name=AAC (Advanced Audio Coding)
profile=LC
codec_type=audio
codec_time_base=1/44100
codec_tag_string=mp4a
codec_tag=0x6134706d
sample_fmt=fltp
sample_rate=44100
channels=2
channel_layout=stereo
bits_per_sample=0
id=N/A
r_frame_rate=0/0avg_frame_rate=0/0
time_base=1/44100
start_pts=0
start_time=0.000000
duration_ts=5135360
duration=116.448073
bit_rate=192015
max_bit_rate=202248
bits_per_raw_sample=N/A
nb_frames=5015
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=1
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
TAG:creation_time=2015-02-12 21:50:26
TAG:language=eng
TAG:handler_name=IsoMedia File Produced by Google, 5-11-2011
[/STREAM]
[FORMAT]
filename=http://jmrcubed.com/cdn_v07/sdb/demo/mp4av/nasa2.mp4
nb_streams=2
nb_programs=0
format_name=mov,mp4,m4a,3gp,3g2,mj2
format_long_name=QuickTime / MOV
start_time=0.000000
duration=116.446667
size=39625399
bit_rate=2722303
probe_score=100
TAG:major_brand=mp42
TAG:minor_version=0
TAG:compatible_brands=isommp42
TAG:creation_time=2015-02-12 21:50:26
[/FORMAT]

For MP4 from TX2’s GStreamer, I got:

[STREAM]
index=0
codec_name=hevc
codec_long_name=HEVC (High Efficiency Video Coding)
profile=Main
codec_type=video
codec_time_base=1/90000
codec_tag_string=[36][0][0][0]
codec_tag=0x0024
width=1280
height=480
coded_width=1280
coded_height=480
has_b_frames=0
sample_aspect_ratio=0:1
display_aspect_ratio=0:1
pix_fmt=yuv420p
level=120
color_range=tv
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=unspecified
timecode=N/A
refs=1
id=0x41
r_frame_rate=30/1
avg_frame_rate=30/1
time_base=1/90000
start_pts=324000000
start_time=3600.000000
duration_ts=3324000
duration=36.933333
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
[/STREAM]
[STREAM]
index=1
codec_name=aac
codec_long_name=AAC (Advanced Audio Coding)
profile=LC
codec_type=audio
codec_time_base=1/48000
codec_tag_string=[15][0][0][0]
codec_tag=0x000f
sample_fmt=fltp
sample_rate=48000
channels=2
channel_layout=stereo
bits_per_sample=0
id=0x42
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/90000
start_pts=324000000

start_time=3600.000000
duration_ts=3327359
duration=36.970656
bit_rate=36375
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
[/STREAM]
[FORMAT]
filename=http://jmrcubed.com/cdn_v07/sdb/demo/mp4av/2cam_640x480.mp4
nb_streams=2
nb_programs=1
format_name=mpegts
format_long_name=MPEG-TS (MPEG-2 Transport Stream)
start_time=3600.000000
duration=36.970656
size=19347832
bit_rate=4186635
probe_score=100
[/FORMAT]

Here is the GStreamer command I was using to generate the MP4 video (two cameras on a customized CSI board with DMICs):

gst-launch-1.0 nvcamerasrc sensor-id=1  \
! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12' \
        ! nvvidconv flip-method=4 \
        ! video/x-raw, width=640, height=480, framerate=30/1, format=I420 \
        ! textoverlay text='camLeft' \
        ! mixer.sink_0 \
         nvcamerasrc sensor-id=0 \
! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12' \
        ! nvvidconv flip-method=4 \
        ! video/x-raw, width=640, height=480, framerate=30/1, format=I420 \
        ! textoverlay text='camRight' \
        ! mixer.sink_1 \
        videomixer name=mixer sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=640 sink_1::ypos=0 \
        ! timeoverlay \
        ! omxh265enc bitrate=4000000 control-rate=2 \
        ! h265parse \
        ! tee name=t t. ! queue \
        ! mpegtsmux name=mux alsasrc 'device=hw:tegrasndt186ref,1' ! audioresample \
        ! audio/x-raw,rate=48000,channels=2 ! queue ! voaacenc bitrate=32000 ! aacparse \
        ! queue ! mux. mux. \
        ! filesink location=2cam_640x480.mp4 -e

By Google search, I found a thread on this forum regarding the issue of playing MP4 on browser,
https://devtalk.nvidia.com/default/topic/1024757/jetson-tx2/tx2-h264-encoded-videos-will-not-play-in-web-browsers/
Apparently the “level” values are different, of which is level=120 in my MP4, versus level=31 in NASA video. The symptom matches what was described in the article.

Assuming the setting of “level” is the culprit, it’s still not clear to me how to fix the problem though. Could anyone elaborate the procedure in further details?
Many thanks in advance!

2

I am far from being “adept” with CODECs (beyond what I’m answering now I won’t be able to answer), but what you are speaking of isn’t about mp4 so much as it is the CODECs your mp4 contains.

FYI, mp4 is not a format so much as it is a container of formats. One thing you will want to do is run a video in vlc to find out what the current mp4 contains. In particular, go to the “Tools” menu, click “Codec Information”. Note that audio and video will each have their own format. Different recording tools will support different CODECs…often you won’t notice options to use different CODECs. Sometimes you will.

How are you creating your video? If you go to Windows and use GeForce Experience, and then use vlc to examine the CODECs, you will find this combination inside of mp4s will work on almost any browser. Some browsers also have optional install of various CODECs, but I suspect the combination of “H264” (MPEG-4 AVC part 10) plus AAC audio will do what you want in high quality.

Some desktop capture software offers different CODECs too even within a single “mp4”…think of mp4 as a bookmark of a set of possible combinations of CODEC and you just got unlucky with your initial combination.

I had to custom build blender to get the CODECs I was interested in because I’m combining GeForce Experience videos (and Open Broadcast Studio captures) and overlaying blender animations on top of it. I haven’t done this with gstreamer, so it is likely just a matter of transcoding formats you can’t use into formats you can use…or recording initially in the format you want.

You might want to look around with this Google search:
[url]linux video transcoding - Google Search
(transcoding implies conversion from one CODEC to another…or to differing sample rates)

3

Please try HLS [url]https://devtalk.nvidia.com/default/topic/992525/[/url]

4

Thanks guys for the input.

I’m generating the video using gstreamer on TX2, and try to play the result mp4 video in different environments. So I have a NASA MP4 video, as a reference for ffprobe showing it works in HTML5 webpage. And I have a TX2 camera shooting at a computer screen to record mp4 video. Trying to play the recorded mp4 on the webpage.

Actually, I just went back to dig up my old video gst code, the result MP4 works on browser. Strange!
I have added the video-only case to the end of the page to demo the result.

Here is the gst code for video only output

gst-launch-1.0 nvcamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv flip-method=4 ! 'video/x-raw, width=720, height=480, framerate=30/1, format=NV12' ! omxh264enc ! 'video/x-h264, streamformat=(string)byte-stream' ! h264parse ! qtmux ! filesink location=hardenc264.mp4 -e

Still don’t know what’s wrong with video + audion gst pipeline though. :\

5

Looks similar to
[url]https://devtalk.nvidia.com/default/topic/1024757/jetson-tx2/tx2-h264-encoded-videos-will-not-play-in-web-browsers/post/5212765/#5212765[/url]
[url]https://devtalk.nvidia.com/default/topic/1031777/jetson-tx1/setting-h264-level-for-omxh264enc/post/5250186/#5250186[/url]

6

@DaneLLL, thanks for the link. Look like the level issue has been resolved in 28.2. I checked my working mp4 video, the level=40. So, it must not be the “level” issue as I had thought.

There could be something iffy in my GST pipeline.

7

Hi tmx3, the NG MP4 is encoded in h265. Is it expected? Not sure if web browser can decode h265.

8

Hi DANELLL, I tried both h265 and h264. H265 may have a some performance issues, which could be a separate topic. I should have used the h264 code for the discussion. Sorry for the distraction.

Now I have narrowed the problem of playable mp4 on browser down to audio pipeline. Without audio, the mp4 output is playable on browser using GST pipeline as below
gst-launch-1.0 nvcamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12’ ! nvvidconv flip-method=4 ! ‘video/x-raw, width=720, height=480, framerate=30/1, format=NV12’ ! omxh264enc ! ‘video/x-h264, streamformat=(string)byte-stream’ ! h264parse ! qtmux ! filesink location=hardenc264.mp4 -e

Then if I add audio device to the pipeline:
gst-launch-1.0 nvcamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12’ ! nvvidconv flip-method=4 ! ‘video/x-raw, width=720, height=480, framerate=30/1, format=NV12’ ! omxh264enc ! ‘video/x-h264, streamformat=(string)byte-stream’ ! mpegtsmux name=mux alsasrc ‘device=hw:1,1’ ! audioresample ! audio/x-raw,rate=48000,channels=2 ! queue ! voaacenc bitrate=32000 ! queue ! mux. mux. ! filesink location=hardenc264_audio.mp4 sync=true async=false qos=true -e

The output mp4 would fail to play on browser.

9

Looks not an issue of NVIDIA omxh264enc. You probably can go to http://gstreamer-devel.966125.n4.nabble.com/

10

Use vlc on the video which fails and the video which works…note the audio CODEC used (the tools have the ability to tell you this information).

You’ll want to change the Jetson side from its current audio CODEC to the one in the working stream. Often the choice depends on licensing issues, but some software may just be too old to have updated or the other CODEC might be an optional CODEC not yet installed.

11

@DaneLLL, thanks for the link, I’ll take a look at the gst forum.

@Linuxdev, it’s a good point. Follow the thought, I ran vlc and did a comparison of NASA video and the mp4 I generated (which is not playable on browser).
http://www.jmrcubed.com/cdn_v07/sdb/demo/mp4av/nasa2.mp4
http://www.jmrcubed.com/cdn_v07/sdb/demo/mp4av/ov_2cam.mp4

They look very similar. But the NASA mp4 has an entry “Frame rate: 30”, and my mp4 doesn’t.
NASA’s video codec: H264-MPEG-4-AVC (part 10)(av1)
My mp4 video codec: H264-MPEG-4AVC (part 10)(h264)
Audio wise, both codec are MPEG AAC (mp4a)
The audio sample rate are slightly different, 48000 Hz vs 44100 Hz.

I wonder how to set the frame rate in the container (the video rate has been set in gst). Could this be critical?

12

I don’t know enough about that, but there are many devices which could refuse to play something where full knowledge of frame rate and other details are not known. In the case of GeForce Experience the format is as portable as you can get it, yet quality is good:
Video:

H264 - MPEG-4 AVC (part 10) (<b>avc1</b>)
30.008142 Hz (this is actually imported as 30Hz, but there is also a standard for something
around 29.97Hz and is not a floating point rounding error...it truly is very very close to 30Hz
but not exactly 30Hz...30.008142 is considered 30Hz, 29.97 is its own separate frame rate).

Audio:

MPEG AAC Audio (mp4a)
48 KHz

Unfortunately I don’t know enough about gstreamer to set the different CODECs or specify frame rates. Eventually I have to learn that, but so far have not.

Does anyone know the difference between the two of these, and how he might control this attribute with gstreamer? Or how to explicitly specify 30Hz frame rate?

H264 - MPEG-4 AVC (part 10) (avc1) # Good
H264-MPEG-4AVC (part 10)(h264) # Bad
13

Thanks linuxdev for all your efforts to help.

I just got the streamer pipeline to work. Here is a command that produces a working mp4 playable on web browsers:

$ gst-launch-1.0 -e \
    nvcamerasrc sensor-id=0 \
  ! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12' \
  ! nvvidconv flip-method=4 \
  ! 'video/x-raw, width=720, height=480, framerate=30/1, format=NV12' \
  ! omxh264enc \
  ! 'video/x-h264, stream-format=(string)byte-stream' \
  ! h264parse \
  ! mux. \
    alsasrc 'device=hw:1,1' \
  ! audioresample \
  ! 'audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)2' \
  ! voaacenc \
  ! mux. \
    mp4mux name=mux ! filesink location=1cam_av.mp4

The result can be seen on demo page: http://www.jmrcubed.com/cdn_v07/sdb/demo/mp4av/index.htm

I’ll consider the case closed, however I don’t see the link “accept the answer” on top of each post. By the way, there is a pending issue about playing the mp4 video on VLC (on TX2 only), I’ll open a separate thread for it.