Error recording and saving mp4 video segments (pipeline manipulation) using nvidia tx2 onboard camer...

Marius_I · October 17, 2018, 8:16am

Hello Guys,
I need to capture (record and save to file) from nvidia tx2 on-board csi camera or an Usb camera connected to Nvidia, Video segments of 5 seconds without losing Frames between segments using Hardware accelerated Encoding (OMXH264ENC in this case ) and recording at a high rate for ex: 1280x720(HD) 120fps.Videos will be saved in MP4 media Format.

System: Nvidia TX2 Ubuntu 16.04.5 LTS , Gstreamer 1.0 (bad,good,ugly plugins) and Python3
Hw: nvidia 5mp Onboard csi camera, Microsoft USB cam;

Solution Found:

Creating and playing a main Pipeline (src) containing camera config with a tee:
Creating a second Pipeline (sink) containing the Encoding part and save-to-file Details:

The idea is to keep playing the Pipeline (main) and add/remove repeatedly the recordpipe every 5 seconds.

The problem is that after a while of addind and removing the second(sometimes after 3 segments or after 20 segments ) the main Pipeline freezes and get block without capturing anything while the second recordpipe is working ok.

Please find the code attached (seg.txt)

Sometimes the main pipeline does not starts , sometimes after 20 segments the main pipeline is getting blocked.

I am open to any solution writen in python.

DaneLLL · October 17, 2018, 8:55am

Hi Marius,
We don’t have much experience to run gstreamer in python. Please give more guidnace. So firstly, we have to update to Python3.0 and then run

$ python seg.txt

Is it correct?

Marius_I · October 17, 2018, 11:01am

Hello DaneLLL,

Just change the Extension from “txt” to “py” and run in command line $ python3 seg.py

Then you should observe the warnings from Encoder (BlockType = 4) when the Pipeline main and recordpipe second started and the segments are recorded and saved to file.

As i was reading the Encoding warning are ok at this Point.

carolyuu · October 18, 2018, 2:47am

Hi Marius_I,

I run below command with python and python3, the recording are work on r28.2.1/TX2.

nvidia@tegra-ubuntu:~$ python seg.py 

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
02:39:46.557920

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 2 WxH = 1280x720 FrameRate = 120.000000 ...

Recording 2018-10-18_02.39.51.mp4...
02:39:51.595889
Recording 2018-10-18_02.39.56.mp4...
02:39:56.615614
Recording 2018-10-18_02.40.01.mp4...
02:40:01.619371

nvidia@tegra-ubuntu:~$ python3 seg.py 

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
02:41:44.000323

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 2 WxH = 1280x720 FrameRate = 120.000000 ...

Recording 2018-10-18_02.41.49.mp4...
02:41:49.018655
Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation
NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation
===== MSENC blits (mode: 1) into tiled surfaces =====
Recording 2018-10-18_02.41.54.mp4...
02:41:54.028065
Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation
NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation

Marius_I · October 18, 2018, 7:25am

Hello Carolyuu,

Configure the segments to 5 seconds for example run the code and let it record and save to file for a longer period of time for instance more than 100 times.
You will see that after a while the Pipeline freezes and the Output will look like :

Available Sensor modes :

2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

02:41:44.000323

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 2 WxH = 1280x720 FrameRate = 120.000000 …

Recording 2018-10-18_02.41.49.mp4…

02:41:49.018655

Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4

===== MSENC =====

NvMMLiteBlockCreate : Block : BlockType = 4

NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation

===== MSENC blits (mode: 1) into tiled surfaces =====

Recording 2018-10-18_02.41.54.mp4…

02:41:54.028065

Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4

===== MSENC =====

NvMMLiteBlockCreate : Block : BlockType = 4

NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance violation

NvH264MSEncSetCommonStreamAttribute: LevelIdc conformance Violation

Recording 2018-10-18_02.45.51.mp4… #########(Pipeline freezes and no stream is saved)

02:39:51.595889

Recording 2018-10-18_02.45.56.mp4…

02:39:56.615614

Recording 2018-10-18_02.45.01.mp4…

02:40:01.619371

##########
The behavior changes after a while(no pattern identified ) and the main pipeline frezzes the encoding warning disappears and file is save with no frame but each file saved after this has a standard size of 616bytes.

carolyuu · October 18, 2018, 9:17am

Hi Marius_I,

We can reproduce your issue.
We will investigation this issue and update to you.

DaneLLL · October 19, 2018, 3:00am

Hi Marius,
We also see issue with SW encoder. Seems not specific to HW encoder.

Are you able to use splitmuxsink in your case?
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good/html/gst-plugins-good-plugins-splitmuxsink.html
seg_x264enc.zip (1.44 KB)

Marius_I · October 19, 2018, 3:13pm

Hello DaneLLL,

I was testing the whole day with a single pipeline and splitmuxsink :

pipeline = Gst.parse_launch ((“nvcamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int){}, height=(int){}, format=(string)I420, framerate=(fraction){}/1 ! nvvidconv flip-method={} ! video/x-raw(memory:NVMM), format=(string)I420 ! omxh264enc ! video/x-h264, stream-format=(string)byte-stream ! h264parse ! splitmuxsink location=%02d.mp4 max-size-time={}”).format(config.width, config.height, config.fps, config.flip, config.segments_time))

It seems a good solution but there are 2 things:

the segments are not exactly 5 seconds, every segment has different size from 4.554 to 4.8129 but never 5.I checked if the frames are missing between segments and seems to be ok.
there is a small bug, when i right-click to see the video file properties the framerate is doing some crazy things showing instead of 120fps, 12000 fps or instead of 90fps, 9000 fps but not for all segments. Some are showing the corect number.I numbered the frames in a 12000fps video file and the number of frames are ok. It seems that there is a problem with the metadata or something.

I will test to see if there are no problems if i merge the segments .

DaneLLL · October 22, 2018, 3:13am

Hi Marius,
Can you try to add insert-sps-pps=true and iframeinterval=30?

$ gst-launch-1.0 nvcamerasrc num-buffers=12000 ! 'video/x-raw(memory:NVMM),width=1280,height=720,framerate=120/1' ! omxh264enc <b>insert-sps-pps=true iframeinterval=30</b> ! h264parse ! splitmuxsink location=%02d.mp4 max-size-time=5000000000

The split is always at IDR frames, so it may not exactly be at 5 seconds. Shortening I frame interval should help it be closer to 5 seconds. For high frame rate case, please also execute jetson_clocks.sh to get max performance.

Marius_I · October 22, 2018, 11:59am

Hello DaneLLL,

I have tested with insert-sps-pps=true and iframeinterval=30 and the segments except the first one (4.99666 seconds) are all 4.75000 seconds which is good.

The only issue is again the metadata regarding the fps of the segment recorded.

For example I have recorded 20 segments of 1280x720 120fps and the segment 06 of 20 is showing when right clicked 240fps.

I have un-framed the video segment (4.75 seconds) and I got 570 png pictures which is correct ( 570png / 120 fps = 4.7500 seconds ).

Somewhere is a problem with the metadata I believe.

Marius_I · October 22, 2018, 12:04pm

To be more specifically regarding details from above, I have recorded 21 video segments and 2 are showing 240fps and one 1200fps.
Do you have an idea why?

DaneLLL · October 22, 2018, 12:20pm

Hi Marius,
Do you use mediainfo MediaInfo ?

If not, please check if you can observe the same with the tool.

Marius_I · October 22, 2018, 1:11pm

DaneLLL,

Thanks for helping.

I have installed MediaInfo and have recorded more video segments.

With mediainfo I am able to see the real fps and is fluctuating for every segment (from 111 to 120fsp).

For those video segments who are showing 1200fps at file properties, mediainfo shows 112 or 117 fps. But this is not a pattern because there are segments which file properties shows 120fps and mediainfo display 117fps.

Conclusion:

mediainfo shows the same information for all video segments. Just ubuntu system file properties is showing incorrect fsp value for some video files.

DaneLLL · October 23, 2018, 2:40am

Hi Marius,
Mediainfo should be more reliable.
Can you also try exiftool?

nvidia@tegra-ubuntu:~$ exiftool 00.mp4
ExifTool Version Number         : 10.10
File Name                       : 00.mp4
Directory                       : .
File Size                       : 2.5 MB
File Modification Date/Time     : 2018:10:23 02:36:12+00:00
File Access Date/Time           : 2018:10:23 02:35:54+00:00
File Inode Change Date/Time     : 2018:10:23 02:36:12+00:00
File Permissions                : rw-rw-r--
File Type                       : MP4
File Type Extension             : mp4
MIME Type                       : video/mp4
Major Brand                     : MP4 v2 [ISO 14496-14]
Minor Version                   : 0.0.0
Compatible Brands               : mp42, mp41, isom, iso2
Movie Data Size                 : 2574467
Movie Data Offset               : 48
Movie Header Version            : 0
Create Date                     : 2018:10:23 02:36:06
Modify Date                     : 2018:10:23 02:36:06
Time Scale                      : 1800
Duration                        : 5.00 s
Preferred Rate                  : 1
Preferred Volume                : 100.00%
Preview Time                    : 0 s
Preview Duration                : 0 s
Poster Time                     : 0 s
Selection Time                  : 0 s
Selection Duration              : 0 s
Current Time                    : 0 s
Next Track ID                   : 2
Track Header Version            : 0
Track Create Date               : 2018:10:23 02:36:06
Track Modify Date               : 2018:10:23 02:36:06
Track ID                        : 1
Track Duration                  : 5.00 s
Track Layer                     : 0
Track Volume                    : 0.00%
Matrix Structure                : 1 0 0 0 1 0 0 0 1
Image Width                     : 1280
Image Height                    : 720
Media Header Version            : 0
Media Create Date               : 2018:10:23 02:36:06
Media Modify Date               : 2018:10:23 02:36:06
Media Time Scale                : 1200
Media Duration                  : 5.00 s
Handler Description             : VideoHandler
Graphics Mode                   : srcCopy
Op Color                        : 0 0 0
Compressor ID                   : avc1
Source Image Width              : 1280
Source Image Height             : 720
X Resolution                    : 72
Y Resolution                    : 72
Bit Depth                       : 24
Buffer Size                     : 0
Max Bitrate                     : 75590403
Average Bitrate                 : 4179531
<b>Video Frame Rate                : 120.06</b>
Handler Class                   : Media Handler
Handler Type                    : Metadata
Avg Bitrate                     : 4.12 Mbps
Image Size                      : 1280x720
Megapixels                      : 0.922
Rotation                        : 0

Video Frame Rate : 120.06

Besides, if ubuntu system file properties show abnormal information for certain video files, are the video files played in correct frame rate and duration(5 second)?

Marius_I · October 23, 2018, 6:47am

Hello DaneLLL,

I have recorded 10 new video segments and for the video segment 09 system file properties along with mplayer shows 1200fps.

The mediainfo and exiftool shows fps 116.588 and 4.86 seconds (is the only one bellow 120fps from all 10 videos)

I have copied the video segment to windows 7 and windows file system properties shows a correct 116fps.

The video file is playable in windows and android having 4.86 seconds and 116fps.

I have tested at lower frame rate of 1280x720 with 90 fps and everything looks fine. A perfect 90.000 for all videos and not oscillating.
Also I have tested with

2592x1944 at 30fps - result 30.000 fps for all videos and no problem;
1920x1080 at 30 fps - result 30.000 fps no problem.

I believe that when recording for a long period of time at a high frame rate event (HD-120fps) with jetson clocks running the pipeline recording is oscillating and sometimes fails to record at exactly 120fps and medatada is affected somehow for Ubuntu file system.

But I managed to merge all the mp4 files and the results for a continuous 100 seconds recording(20segments) looks good with only 0.126 seconds being lost for HD-120fps.

Thank you for helping me!

Marius_I · October 23, 2018, 12:45pm

Hello,

Does anybody know regarding “splitmuxsink location=%02d.mp4” how to put instead of decimal values 01.mp4 ,02.mp4 timestampe for each video file segment?

It has to do with the %02d special char but I cannot find a solution for putting timestamps as name.

I tried all the combination like “splitmuxsink location=%d%ti.mp4” but still no result.

Do you guys have an idea?

DaneLLL · October 24, 2018, 3:20am

Hi Marius,
Please try

splitmuxsink location=test-$( date +%T )-%02d.mp4

It is pure gstreamer issue and you can also go to http://gstreamer-devel.966125.n4.nabble.com/ to get help.

Marius_I · October 24, 2018, 12:25pm

Hello DaneLLL,

I know that is a gstreamer issue and not NVidia but I have searched on their forum and got nothing.
On gstreamer forum there are solutions only for c++ .

Also I tried your solution but says that ( date ) is not an element.
I do not know from where is date taken.
Where have you found that solution?

DaneLLL · October 25, 2018, 5:15am

Hi Marius,
I run single pipeline:

$ gst-launch-1.0 nvcamerasrc num-buffers=12000 ! 'video/x-raw(memory:NVMM),width=1280,height=720,framerate=120/1' ! omxh264enc insert-sps-pps=true iframeinterval=30 ! h264parse ! splitmuxsink location=test-$( date +%T )-%02d.mp4 max-size-time=5000000000

I refer to the link GStreamer - LinuxTVWiki
It shows

filesink location=test-$( date --iso-8601=seconds ).avi

But it may not work for python.

Marius_I · November 12, 2018, 8:27am

Solution for renaming the recorded MP4 video segments with timestamp when using splitmuxsink in python(same for c++).

when running command : gst-inspect-1.0 splitmuxsink at the bottom under the element properties there is a :

Element Signals:
“format-location” : gchararray user_function (GstElement* object,
guint arg0,
gpointer user_data);
The steps are:

create the function callback with 2 parameters and a string Return type which is the name of each videofile.
give a name to the splitmuxsink element in the pipeline
get from the pipeline by name that splitmuxsink element
connect to the bus using that name and also the function to be calledback
What we have done we have created a signal for that splitmuxsink element.

Topic		Replies	Views
Capture raw video through Gstreamer with CSI cameras Jetson TX2	38	17569	October 18, 2021
using gstreamer to transcode/downscale 1080i videos on Jetson TX2 Jetson TX2	19	3856	October 18, 2021
Gstreamer issue in NX: Nano is faster than NX! Jetson Xavier NX gstreamer	16	1989	October 18, 2021
I am facing issue with splitmuxsink in deepstream DeepStream SDK gstreamer , deepstream	13	37	March 7, 2025
Simple Gstream USB Camera Live View Jetson AGX Orin gstreamer	12	3106	July 25, 2023
Gstreamer reports Raspberry Pi camera streaming at 120fps when in reality it is only 60fps Jetson Nano camera , gstreamer	53	3833	July 19, 2022
Video and Audio Inputs Test cmd Jetson TX2	15	4879	January 30, 2018
Frame Drop and Latency Issues in DeepStream GStreamer Pipeline DeepStream SDK cuda , gstreamer , jetson , deepstream , jetson-nano , deepstream61	12	138	March 31, 2025
omxh264enc does not send keyframes at regular intervals Jetson TX1	23	3796	October 18, 2021
recording video stream from usb cameras Jetson Nano	13	4801	October 18, 2021

Error recording and saving mp4 video segments (pipeline manipulation) using nvidia tx2 onboard camer...

Related topics