Now I want to decode local mp4 video files and rtsp stream on Jetson Xavier NX on background , in other words , it can’t appear a surface to show video frame liked OSD.
I try to create a playbin pipeline and appsink to do this(by callback on_new_sample) , however , it’s decode by cpu , and have no used nvv4l2decoder hardware accelerated , if video decoded by software , it will affect my system performace.How can I do this , please give me some suggestion.Thanks.
Hi,
You can try uridecodebin. Please refer to
Cannot get the playback rtsp stream for deepstream yolo and do inference - #9 by DaneLLL
For eliminating display output, you may replace nvoverlaysink with fakesink
Hi ,if i use uridecodebin to decode ,there are flowing issues bellow:
- How can i capture the rgba frame when one frame docode complete?If I appand a fakesink probe callback i don’t know how to attach the rgba frame.
- How to replay the local file when get an eos signal? If I use playbin I can seek to the beginning frame , however I don’t know how to complete this function by uridocodebin.
Hi,
We are able to use hardware decoder in running playbin. We run the commands and see nvv4l2decoder being picked and run:
$ export GST_DEBUG=*FACTORY*:4
$ gst-launch-1.0 playbin uri= file:///opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_720p.mp4
You may check why you don’t see hardware decoder in your usecase.
hi , Thanks your reply. I run the same commands as you said , the log ouput liked this:
nvidia@nvidia-desktop:~$ export GST_DEBUG=FACTORY:4
nvidia@nvidia-desktop:~$ gst-launch-1.0 playbin uri= file:///opt/nvidia/deepstream/deepstream-5.0/samples/streams/sample_720p.mp4
0:00:00.071255924 31989 0x558a35b800 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “playbin”
Setting pipeline to PAUSED …
0:00:00.073134519 31989 0x558a35b800 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “uridecodebin”
0:00:00.075337360 31989 0x558a35b800 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “filesrc” named “source”
0:00:00.075854996 31989 0x558a35b800 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “decodebin”
0:00:00.076194827 31989 0x558a35b800 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “typefind” named “typefind”
Pipeline is PREROLLING …
0:00:00.131655756 31989 0x558a36ce30 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “qtdemux”
0:00:00.135245640 31989 0x7f941bb2d0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “multiqueue”
0:00:00.140301514 31989 0x7f941bb2d0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “h264parse”
0:00:00.142077926 31989 0x7f941bb2d0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “capsfilter”
0:00:00.151195878 31989 0x7f941bb2d0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “aacparse”
0:00:00.162564707 31989 0x7f941bb2d0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “pulsesink”
0:00:02.591390420 31989 0x7f941bb2d0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “avdec_aac”
0:00:04.763718206 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “nvoverlaysink”
0:00:04.777550186 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “nvv4l2decoder”
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
0:00:04.950642972 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “input-selector”
0:00:04.951684807 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “input-selector”
0:00:04.952843963 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “tee” named “audiotee”
0:00:04.955330672 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “bin” named “vbin”
0:00:04.955746382 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “queue” named “vqueue”
0:00:04.956638607 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “identity” named “identity”
0:00:04.958070871 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “videobalance” named “videobalance”
0:00:04.959145509 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “videoconvert” named “conv”
0:00:04.960498919 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “videoscale” named “scale”
0:00:04.975050727 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “videoconvert” named “conv”
0:00:04.975274839 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “videoscale” named “scale”
0:00:04.975511720 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “videoconvert” named “conv2”
0:00:05.010569240 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “bin” named “vdbin”
0:00:05.010675040 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “videoconvert” named “vdconv”
0:00:05.013352098 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “deinterlace” named “deinterlace”
0:00:05.016053094 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “bin” named “abin”
0:00:05.016177071 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “queue” named “aqueue”
0:00:05.016636753 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “identity” named “identity”
0:00:05.017594358 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “volume” named “volume”
0:00:05.018515897 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “audioconvert” named “conv”
0:00:05.019445853 31989 0x7f88014590 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “audioresample” named “resample”
Redistribute latency…
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
New clock: GstPulseSinkClock
Got EOS from element “playbin0”.
Execution ended after 0:00:48.074204498
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …
As logs above show , It seems that nvv4l2decoder have been picked and run ,however I run jtop command I will see the consumption rate is alway 0% about the GPU items when video playing. If it’s normal?
Hi,
The decoding is done on NVDEC engine. Please run sudo tegrastats and check if the engine is enabled.
Hi，when video decoding I run sudo tegrastats show info：
RAM 1988/7774MB (lfb 80x4MB) SWAP 0/3887MB (cached 0MB) CPU [3%@1190,1%@1190,1%@1190,1%@1190,0%@1190,1%@1343] EMC_FREQ 1%@1600 GR3D_FREQ 0%@114 NVDEC 128 NVDEC1 128 VIC_FREQ 0%@115 APE 150 MTS fg 0% bg 2% AO@43C GPU@42.5C PMIC@100C AUX@44C CPU@42.5C thermal@43.1C VDD_IN 3535/3394 VDD_CPU_GPU_CV 365/315 VDD_SOC 1217/1156
I’m not sure if NVDEC engine is enabled.
If I use uridecodebin to decode the status is liked this:
RAM 2072/7774MB (lfb 78x4MB) SWAP 0/3887MB (cached 0MB) CPU [69%@1204,36%@1418,5%@1419,7%@1419,2%@1419,6%@1419] EMC_FREQ 9%@1600 GR3D_FREQ 57%@114 NVDEC 153 NVDEC1 153 VIC_FREQ 0%@115 APE 150 MTS fg 0% bg 3% AO@42C GPU@42.5C PMIC@100C AUX@43.5C CPU@42C thermal@42.4C VDD_IN 4640/4079 VDD_CPU_GPU_CV 1097/917 VDD_SOC 1381/1203
RAM 2071/7774MB (lfb 78x4MB) SWAP 0/3887MB (cached 0MB) CPU [24%@1413,38%@1224,17%@1408,40%@1410,9%@1410,11%@1410] EMC_FREQ 9%@1600 GR3D_FREQ 11%@114 NVDEC 153 NVDEC1 153 VIC_FREQ 90%@115 APE 150 MTS fg 0% bg 6% AO@42C GPU@42C PMIC@100C AUX@43C CPU@42.5C thermal@42.4C VDD_IN 4680/4081 VDD_CPU_GPU_CV 1097/917 VDD_SOC 1381/1203
So. it’s obviously that uridecodebin use NVDEC engin becase of the GR3D_FREQ info item is not zero , it’s liked this “GR3D_FREQ 57%@114 NVDEC” mean that 57% used. However playbin is always 0%.
Hi,
Your understanding is not correct. If hardware decoding is not run, you will not see NVENC and NVENC1. So hardware decoding is enabled in both uridecodebin and playbin. The GPU loading is from other plugin(maybe nveglglessink).
OK, how to replay local mp4 files by uridecodebin?
Please try
$ gst-launch-1.0 uridecodebin uri=file:///_PATH_TO_VIDEO_FILE_ ! nvoverlaysink
Hi Dane ,
I mean how to play local mp4 files infinite loop by uridecodebin.I will take the deepstream_test3_app sample case for example. When I get a GST_MESSAGE_EOS signal in bus_call, I need to replay the video instead of quitting.
Hi,
Please check if this works:
GStreamer Infinite Loop Version 2 · GitHub
Send GST_SEEK_FLAG_FLUSH if you get EoS signal:
pipeline.seek_simple(Gst.Format.TIME, Gst.SeekFlags.FLUSH, 0)
I do this by flowing codes, but it don’t work.The parameter passed is the source GstBin object create by “gst_bin_new”.
static gboolean
seek_decode (gpointer data)
{
GstElement *bin = (GstElement *) data;
gboolean ret = TRUE;
gst_element_set_state (bin, GST_STATE_PAUSED);
ret = gst_element_seek (bin, 1.0, GST_FORMAT_TIME,
(GstSeekFlags) (GST_SEEK_FLAG_KEY_UNIT | GST_SEEK_FLAG_FLUSH),
GST_SEEK_TYPE_SET, 0, GST_SEEK_TYPE_NONE, GST_CLOCK_TIME_NONE);
if (!ret)
GST_WARNING (“Error in seeking pipeline”);
gst_element_set_state (bin, GST_STATE_PLAYING);
return FALSE;
}
The detail error info show:
ERROR from element qtdemux0: Internal data stream error.
Error details: qtdemux.c(6073): gst_qtdemux_loop (): /GstPipeline:dstest3-pipeline/GstBin:source-bin-00/GstURIDecodeBin:uri-decode-bin/GstDecodeBin:decodebin0/GstQTDemux:qtdemux0:
streaming stopped, reason error (-5)
Returned, stopping playback
Deleting pipeline
Hi,
A possible solution is to run the pipeline:
$ gst-launch-1.0 multifilesrc location= test_%02d.ts loop=1 ! tsdemux ! h264parse ! nvv4l2decoder ! mx.sink_0 nvstreammux width=1920 height=1080 batch-size=1 name=mx ! nvinfer config-file-path=/home/nvidia/deepstream-5.1/samples/configs/deepstream-app/config_infer_primary_nano.txt unique-id=7 ! nvinfer config-file-path=/home/nvidia/deepstream-5.1/samples/configs/deepstream-app/config_infer_primary.txt unique-id=8 ! nvvideoconvert ! nvdsosd ! nvoverlaysink
The source file has to be ts or mkv and have index in the file name(such as test_00.ts). If your files is mp4, you would need to re-mux it into ts or mkv.