Error recording and saving mp4 video segments (pipeline manipulation) using nvidia tx2 onboard camer...

Hello Guys,
I need to capture (record and save to file) from nvidia tx2 on-board csi camera or an Usb camera connected to Nvidia, Video segments of 5 seconds without losing Frames between segments using Hardware accelerated Encoding (OMXH264ENC in this case ) and recording at a high rate for ex: 1280x720(HD) 120fps.Videos will be saved in MP4 media Format.

System: Nvidia TX2 Ubuntu 16.04.5 LTS , Gstreamer 1.0 (bad,good,ugly plugins) and Python3
Hw: nvidia 5mp Onboard csi camera, Microsoft USB cam;

Solution Found:

  • Creating and playing a main Pipeline (src) containing camera config with a tee:
  • Creating a second Pipeline (sink) containing the Encoding part and save-to-file Details:

The idea is to keep playing the Pipeline (main) and add/remove repeatedly the recordpipe every 5 seconds.

The problem is that after a while of addind and removing the second(sometimes after 3 segments or after 20 segments ) the main Pipeline freezes and get block without capturing anything while the second recordpipe is working ok.

Please find the code attached (seg.txt)

Sometimes the main pipeline does not starts , sometimes after 20 segments the main pipeline is getting blocked.

I am open to any solution writen in python.

Hi Marius,
We don’t have much experience to run gstreamer in python. Please give more guidnace. So firstly, we have to update to Python3.0 and then run

$ python seg.txt

Is it correct?

Hello DaneLLL,

Just change the Extension from “txt” to “py” and run in command line $ python3 seg.py

Then you should observe the warnings from Encoder (BlockType = 4) when the Pipeline main and recordpipe second started and the segments are recorded and saved to file.

As i was reading the Encoding warning are ok at this Point.

Hi Marius_I,

I run below command with python and python3, the recording are work on r28.2.1/TX2.

nvidia@tegra-ubuntu:~$ python seg.py 

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
02:39:46.557920

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 2 WxH = 1280x720 FrameRate = 120.000000 ...

Recording 2018-10-18_02.39.51.mp4...
02:39:51.595889
Recording 2018-10-18_02.39.56.mp4...
02:39:56.615614
Recording 2018-10-18_02.40.01.mp4...
02:40:01.619371
nvidia@tegra-ubuntu:~$ python3 seg.py 

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
02:41:44.000323

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 2 WxH = 1280x720 FrameRate = 120.000000 ...

Recording 2018-10-18_02.41.49.mp4...
02:41:49.018655
Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation
NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation
===== MSENC blits (mode: 1) into tiled surfaces =====
Recording 2018-10-18_02.41.54.mp4...
02:41:54.028065
Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation
NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation

Hello Carolyuu,

Configure the segments to 5 seconds for example run the code and let it record and save to file for a longer period of time for instance more than 100 times.
You will see that after a while the Pipeline freezes and the Output will look like :

Available Sensor modes :

2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

02:41:44.000323

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 2 WxH = 1280x720 FrameRate = 120.000000 …

Recording 2018-10-18_02.41.49.mp4…

02:41:49.018655

Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4

===== MSENC =====

NvMMLiteBlockCreate : Block : BlockType = 4

NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation

NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation

===== MSENC blits (mode: 1) into tiled surfaces =====

Recording 2018-10-18_02.41.54.mp4…

02:41:54.028065

Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4

===== MSENC =====

NvMMLiteBlockCreate : Block : BlockType = 4

NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation

NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance Violation

Recording 2018-10-18_02.45.51.mp4… #########(Pipeline freezes and no stream is saved)

02:39:51.595889

Recording 2018-10-18_02.45.56.mp4…

02:39:56.615614

Recording 2018-10-18_02.45.01.mp4…

02:40:01.619371

##########
The behavior changes after a while(no pattern identified ) and the main pipeline frezzes the encoding warning disappears and file is save with no frame but each file saved after this has a standard size of 616bytes.

Hi Marius_I,

We can reproduce your issue.
We will investigation this issue and update to you.

Hi Marius,
We also see issue with SW encoder. Seems not specific to HW encoder.

Are you able to use splitmuxsink in your case?
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good/html/gst-plugins-good-plugins-splitmuxsink.html
seg_x264enc.zip (1.44 KB)

Hello DaneLLL,

I was testing the whole day with a single pipeline and splitmuxsink :

pipeline = Gst.parse_launch ((“nvcamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int){}, height=(int){}, format=(string)I420, framerate=(fraction){}/1 ! nvvidconv flip-method={} ! video/x-raw(memory:NVMM), format=(string)I420 ! omxh264enc ! video/x-h264, stream-format=(string)byte-stream ! h264parse ! splitmuxsink location=%02d.mp4 max-size-time={}”).format(config.width, config.height, config.fps, config.flip, config.segments_time))

It seems a good solution but there are 2 things:

  1. the segments are not exactly 5 seconds, every segment has different size from 4.554 to 4.8129 but never 5.I checked if the frames are missing between segments and seems to be ok.
  2. there is a small bug, when i right-click to see the video file properties the framerate is doing some crazy things showing instead of 120fps, 12000 fps or instead of 90fps, 9000 fps but not for all segments. Some are showing the corect number.I numbered the frames in a 12000fps video file and the number of frames are ok. It seems that there is a problem with the metadata or something.

I will test to see if there are no problems if i merge the segments .

Hi Marius,
Can you try to add insert-sps-pps=true and iframeinterval=30?

$ gst-launch-1.0 nvcamerasrc num-buffers=12000 ! 'video/x-raw(memory:NVMM),width=1280,height=720,framerate=120/1' ! omxh264enc <b>insert-sps-pps=true iframeinterval=30</b> ! h264parse ! splitmuxsink location=%02d.mp4 max-size-time=5000000000

The split is always at IDR frames, so it may not exactly be at 5 seconds. Shortening I frame interval should help it be closer to 5 seconds. For high frame rate case, please also execute jetson_clocks.sh to get max performance.

Hello DaneLLL,

I have tested with insert-sps-pps=true and iframeinterval=30 and the segments except the first one (4.99666 seconds) are all 4.75000 seconds which is good.

The only issue is again the metadata regarding the fps of the segment recorded.

For example I have recorded 20 segments of 1280x720 120fps and the segment 06 of 20 is showing when right clicked 240fps.

I have un-framed the video segment (4.75 seconds) and I got 570 png pictures which is correct ( 570png / 120 fps = 4.7500 seconds ).

Somewhere is a problem with the metadata I believe.

To be more specifically regarding details from above, I have recorded 21 video segments and 2 are showing 240fps and one 1200fps.
Do you have an idea why?

Hi Marius,
Do you use mediainfo https://mediaarea.net/en/MediaInfo ?

If not, please check if you can observe the same with the tool.

DaneLLL,

Thanks for helping.

I have installed MediaInfo and have recorded more video segments.

With mediainfo I am able to see the real fps and is fluctuating for every segment (from 111 to 120fsp).

For those video segments who are showing 1200fps at file properties, mediainfo shows 112 or 117 fps. But this is not a pattern because there are segments which file properties shows 120fps and mediainfo display 117fps.

Conclusion:

  • mediainfo shows the same information for all video segments. Just ubuntu system file properties is showing incorrect fsp value for some video files.

Hi Marius,
Mediainfo should be more reliable.
Can you also try exiftool?

nvidia@tegra-ubuntu:~$ exiftool 00.mp4
ExifTool Version Number         : 10.10
File Name                       : 00.mp4
Directory                       : .
File Size                       : 2.5 MB
File Modification Date/Time     : 2018:10:23 02:36:12+00:00
File Access Date/Time           : 2018:10:23 02:35:54+00:00
File Inode Change Date/Time     : 2018:10:23 02:36:12+00:00
File Permissions                : rw-rw-r--
File Type                       : MP4
File Type Extension             : mp4
MIME Type                       : video/mp4
Major Brand                     : MP4 v2 [ISO 14496-14]
Minor Version                   : 0.0.0
Compatible Brands               : mp42, mp41, isom, iso2
Movie Data Size                 : 2574467
Movie Data Offset               : 48
Movie Header Version            : 0
Create Date                     : 2018:10:23 02:36:06
Modify Date                     : 2018:10:23 02:36:06
Time Scale                      : 1800
Duration                        : 5.00 s
Preferred Rate                  : 1
Preferred Volume                : 100.00%
Preview Time                    : 0 s
Preview Duration                : 0 s
Poster Time                     : 0 s
Selection Time                  : 0 s
Selection Duration              : 0 s
Current Time                    : 0 s
Next Track ID                   : 2
Track Header Version            : 0
Track Create Date               : 2018:10:23 02:36:06
Track Modify Date               : 2018:10:23 02:36:06
Track ID                        : 1
Track Duration                  : 5.00 s
Track Layer                     : 0
Track Volume                    : 0.00%
Matrix Structure                : 1 0 0 0 1 0 0 0 1
Image Width                     : 1280
Image Height                    : 720
Media Header Version            : 0
Media Create Date               : 2018:10:23 02:36:06
Media Modify Date               : 2018:10:23 02:36:06
Media Time Scale                : 1200
Media Duration                  : 5.00 s
Handler Description             : VideoHandler
Graphics Mode                   : srcCopy
Op Color                        : 0 0 0
Compressor ID                   : avc1
Source Image Width              : 1280
Source Image Height             : 720
X Resolution                    : 72
Y Resolution                    : 72
Bit Depth                       : 24
Buffer Size                     : 0
Max Bitrate                     : 75590403
Average Bitrate                 : 4179531
<b>Video Frame Rate                : 120.06</b>
Handler Class                   : Media Handler
Handler Type                    : Metadata
Avg Bitrate                     : 4.12 Mbps
Image Size                      : 1280x720
Megapixels                      : 0.922
Rotation                        : 0

Video Frame Rate : 120.06

Besides, if ubuntu system file properties show abnormal information for certain video files, are the video files played in correct frame rate and duration(5 second)?

Hello DaneLLL,

I have recorded 10 new video segments and for the video segment 09 system file properties along with mplayer shows 1200fps.

The mediainfo and exiftool shows fps 116.588 and 4.86 seconds (is the only one bellow 120fps from all 10 videos)

I have copied the video segment to windows 7 and windows file system properties shows a correct 116fps.

The video file is playable in windows and android having 4.86 seconds and 116fps.

I have tested at lower frame rate of 1280x720 with 90 fps and everything looks fine. A perfect 90.000 for all videos and not oscillating.
Also I have tested with

  • 2592x1944 at 30fps - result 30.000 fps for all videos and no problem;
  • 1920x1080 at 30 fps - result 30.000 fps no problem.

I believe that when recording for a long period of time at a high frame rate event (HD-120fps) with jetson clocks running the pipeline recording is oscillating and sometimes fails to record at exactly 120fps and medatada is affected somehow for Ubuntu file system.

But I managed to merge all the mp4 files and the results for a continuous 100 seconds recording(20segments) looks good with only 0.126 seconds being lost for HD-120fps.

Thank you for helping me!

Hello,

Does anybody know regarding “splitmuxsink location=%02d.mp4” how to put instead of decimal values 01.mp4 ,02.mp4 timestampe for each video file segment?

It has to do with the %02d special char but I cannot find a solution for putting timestamps as name.

I tried all the combination like “splitmuxsink location=%d%ti.mp4” but still no result.

Do you guys have an idea?

Hi Marius,
Please try

splitmuxsink location=test-$( date +%T )-%02d.mp4

It is pure gstreamer issue and you can also go to http://gstreamer-devel.966125.n4.nabble.com/ to get help.

Hello DaneLLL,

I know that is a gstreamer issue and not NVidia but I have searched on their forum and got nothing.
On gstreamer forum there are solutions only for c++ .

Also I tried your solution but says that ( date ) is not an element.
I do not know from where is date taken.
Where have you found that solution?

Hi Marius,
I run single pipeline:

$ gst-launch-1.0 nvcamerasrc num-buffers=12000 ! 'video/x-raw(memory:NVMM),width=1280,height=720,framerate=120/1' ! omxh264enc insert-sps-pps=true iframeinterval=30 ! h264parse ! splitmuxsink location=test-$( date +%T )-%02d.mp4 max-size-time=5000000000

I refer to the link https://www.linuxtv.org/wiki/index.php/GStreamer
It shows

filesink location=test-$( date --iso-8601=seconds ).avi

But it may not work for python.

Solution for renaming the recorded MP4 video segments with timestamp when using splitmuxsink in python(same for c++).

when running command : gst-inspect-1.0 splitmuxsink at the bottom under the element properties there is a :

Element Signals:
“format-location” : gchararray user_function (GstElement* object,
guint arg0,
gpointer user_data);
The steps are:

  1. create the function callback with 2 parameters and a string Return type which is the name of each videofile.
  2. give a name to the splitmuxsink element in the pipeline
  3. get from the pipeline by name that splitmuxsink element
  4. connect to the bus using that name and also the function to be calledback
    What we have done we have created a signal for that splitmuxsink element.