Wrapping smartrecord API in a plugin fails when adding audio pad

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

• Hardware Platform Jetson
• DeepStream Version 6.0
• JetPack Version 4.6
• Issue Type question

I’m trying to wrap the SmartRecord API in a Gstreamer (Bin) element. It seems to be working alright with video only like:

gst-launch-1.0 v4l2src io-mode=2 do-timestamp=TRUE device=/dev/video1 ! video/x-raw ! queue ! nvvidconv ! "video/x-raw(memory:NVMM),format=I420" ! nvv4l2h264enc maxperf-enable=1 disable-cabac=true idrinterval=60 ! h264parse ! mtdsmartrecord name=smartrec

but when i’m adding an audio ghostpad ( see the #ifdef ADD_AUDIO parts) to the Element it fails completely when trying to finalise (=call NvDsSRStart) the file like (using the same pipeline as above):

0:01:41.608964063   376   0x7f94545c50 WARN                 basesrc gstbasesrc.c:3583:gst_base_src_start_complete:<source> pad not activated yet
0:01:41.610548750   376   0x7f94545c50 WARN                 basesrc gstbasesrc.c:3583:gst_base_src_start_complete:<source> pad not activated yet
0:01:41.631333233   376   0x7f48064e30 WARN                 qtdemux qtdemux_types.c:233:qtdemux_type_get: unknown QuickTime node type pasp
0:01:41.631486728   376   0x7f48064e30 WARN                 qtdemux qtdemux.c:3031:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
0:01:41.631621472   376   0x7f48064e30 WARN                 qtdemux qtdemux.c:3031:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 2
0:01:41.631658973   376   0x7f48064e30 WARN                 qtdemux qtdemux.c:11710:qtdemux_parse_trak:<qtdemux0> error: This file is corrupt and cannot be played.
0:01:41.633203607   376   0x7f94545c50 WARN              discoverer gstdiscoverer.c:1473:handle_message:<qtdemux0> Got an error [debug:qtdemux.c(11710): qtdemux_parse_trak (): /GstPipeline:Discoverer/GstURIDecodeBin:discoverer-uri/GstDecodeBin:decodebin1/GstQTDemux:qtdemux0], [message:This file is corrupt and cannot be played.]
** ERROR: <RunUserCallback:207>: No video stream found
** ERROR: <RunUserCallback:212>: No audio stream found

it does create a file, however it seems to be corrupted because when i look at it :

/data/videos$ ffmpeg -i smartrecord_00000_20220404-021611_376.mp4
ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5584d7b640] Invalid mdhd time scale 0, defaulting to 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5584d7b640] invalid STSD entries 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5584d7b640] error reading header
smartrecord_00000_20220404-021611_376.mp4: Invalid data found when processing input

Can anyone see what I am doing wrong with adding the ghostpad for the audio ?
I’m wondering because if I don’t #define ADD_AUDIO this plugin seems to work alright for the video.

Kind regards,
Tom
gstmtdsmartrecord.cpp (14.0 KB)

gstmtdsmartrecord.h (1.4 KB)

#3

Sorry for the late response, have you managed to get issue resolved or still need the support? Thanks

#4

No worries,
yes still need some support please.

currently I don’t have a way to also get the audio into the smartrecord ‘recordbin’
I have fixed some issues with the plugin so i’ll add the updated one here as well.

the issue is still that i’m not sure how to add an audio_0 pad to the custom bin and connect it up with the recordbin of the smartrecord plugin.
i tried to do the same as what i do for the video pad, but it doesn’t seem to like it.
any help would be appreciated.

gstmtdsmartrecord.cpp (22.8 KB)
gstmtdsmartrecord.h (3.5 KB)

#6

I didn’t meet any error with the attached source code, makefile and pipeline:

gstmtdsmartrecord.cpp (14.3 KB)
gstmtdsmartrecord.h (1.5 KB)
Makefile (2.7 KB)

GST_DEBUG=v4l2videoenc:5 gst-launch-1.0 videotestsrc ! videoconvert ! 'video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1' ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=I420' ! nvv4l2h264enc ! h264parse ! mtdsmartrecord name=smartrec

1 Like
#7

@Fiona.Chen oh wow, that is good.

alright, i’ll do the same here with your Makefile and will slowly make changes to my pipeline and see where it breaks. Thank you for giving me at least a way forward to investigate.

Is it alright to leave this question open for another week when i’m back at work ?

#8

Is this still an issue to support? Any result can be shared? Thanks

1 Like
#9

@kayccc i’m testing it now. see if i can find where it goes wrong for me.

#10

@kayccc thank you for your patients , please feel free to close this issue… it was a very silly mistake in my code that showed up because of missing parameters… i need to specify some defaults for the parameters.

thank you all for your help.

#11

@kayccc @Fiona.Chen would it be a good idea for nvidia to supply a plugin around the API call like i did ?

#12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.