Gstreamer loses frames while encoding usb camera with rtsp camera

Autonomous Machines Jetson & Embedded Systems Jetson Nano
,
#1

Hi,

I am trying to run the below pipeline

gst-launch-1.0 -e v4l2src device=/dev/presentation ! queue ! tee name=t1 ! video/x-raw, width=1920, height=1080, framerate=30/1 ! videorate ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 profile=4 ! queue ! h264parse ! queue ! mux.video_0 liveadder name=mix1 ! queue ! audioconvert ! voaacenc ! mux.audio_0 alsasrc device=“hw:presentationAud,0” ! queue ! audio/x-raw ! queue ! audioresample ! “audio/x-raw,rate=48000,format=S16LE,layout=interleaved,channels=2” ! audioamplify amplification=1 ! tee name=a1 ! queue ! mix1.sink_0 alsasrc device=“hw:presentationAud,1” ! queue ! audio/x-raw ! queue ! audioresample ! “audio/x-raw,rate=48000,format=S16LE,layout=interleaved,channels=2” ! audioamplify amplification=1 ! tee name=a2 ! queue ! mix1.sink_1 mp4mux name=mux ! filesink location=feed1.mp4 rtspsrc location=rtsp://192.168.100.82:554/1 ! application/x-rtp, media=video, encoding-name=H264 ! rtph264depay ! tee name=t2 ! queue ! h264parse ! queue ! mp4mux ! filesink location=rtsp.mp4 videotestsrc pattern=black ! video/x-raw,width=320,height=240 ! nvvidconv ! queue ! comp2.sink_0 t1. ! video/x-raw, width=1920, height=1080, framerate=30/1 ! videorate ! video/x-raw, width=1920, height=1080, framerate=30/1 ! queue ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=960,height=540,format=NV12” ! queue ! comp2.sink_1 t2. ! queue ! h264parse ! queue ! nvv4l2decoder ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=960,height=540,format=NV12” ! queue ! comp2.sink_2 nvcompositor name=comp2 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=0 sink_1::ypos=270 sink_1::width=960 sink_1::height=540 sink_2::xpos=960 sink_2::ypos=270 sink_2::width=960 sink_2::height=540 ! nvvidconv ! nvoverlaysink liveadder name=mix ! queue ! audioconvert ! audioresample ! alsasink device=“hw:0,3” a1. ! queue ! mix.sink_0 a2. ! queue ! mix.sink_1 t1. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! videorate ! video/x-raw, width=1920, height=1080, framerate=1/2 ! jpegenc ! multifilesink location=rtsp1.jpg t2. ! h264parse ! queue ! nvv4l2decoder ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=960,height=540,format=NV12” ! multifilesink location=rtsp2.jpg

The pipeline is working without errors. But the usb camera feed is saved and previewed skipping frames and the RTSP feed save and the preview has no issue.

I tried removing audio, only saving part but still the usb feed is skipping frames rtsp feed has no issue.
Is there a latency issue?
How to solve this issue?

Thanks in advanced

#3

Hi,
You can check if the v4l2 source runs well:

gst-launch-1.0 -ev v4l2src device=/dev/presentation ! queue ! tee name=t1 ! video/x-raw, width=1920, height=1080, framerate=30/1 ! videorate ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! fpsdisplaysink text-overlay=0 video-sink=nvoverlaysink sync=0

If no issue is seen with the pipeline. Probably CPU cores are at full loading and impacts the performance. You can run sudo tegrastats to get system loading.

Hardware converter is also heavy in running the command. Please refer to the topic to fix it at max frequency:
Nvvideoconvert issue, nvvideoconvert in DS4 is better than Ds5? - #3 by DaneLLL

#4

Hi @DaneLLL ,

Thank you for the reply. I was able to solve the issue by adding latency=0 after the rtsp source. The issue was the rtsp source is delayed so the othe source starts to lag.

gst-launch-1.0 -e v4l2src device=/dev/presentation ! queue ! tee name=t1 ! video/x-raw, width=1920, height=1080, framerate=30/1 ! videorate ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 profile=4 ! queue ! h264parse ! queue ! mux.video_0 liveadder name=mix1 ! queue ! audioconvert ! voaacenc ! mux.audio_0 alsasrc device=“hw:presentationAud,0” ! queue ! audio/x-raw ! queue ! audioresample ! “audio/x-raw,rate=48000,format=S16LE,layout=interleaved,channels=2” ! audioamplify amplification=1 ! tee name=a1 ! queue ! mix1.sink_0 alsasrc device=“hw:presentationAud,1” ! queue ! audio/x-raw ! queue ! audioresample ! “audio/x-raw,rate=48000,format=S16LE,layout=interleaved,channels=2” ! audioamplify amplification=1 ! tee name=a2 ! queue ! mix1.sink_1 mp4mux name=mux ! filesink location=feed1.mp4 rtspsrc location=rtsp://192.168.100.82:554/1 latency=0 ! application/x-rtp, media=video, encoding-name=H264 ! rtph264depay ! tee name=t2 ! queue ! h264parse ! queue ! mp4mux ! filesink location=rtsp.mp4 videotestsrc pattern=black ! video/x-raw,width=320,height=240 ! nvvidconv ! queue ! comp2.sink_0 t1. ! video/x-raw, width=1920, height=1080, framerate=30/1 ! videorate ! video/x-raw, width=1920, height=1080, framerate=30/1 ! queue ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=960,height=540,format=NV12” ! queue ! comp2.sink_1 t2. ! queue ! h264parse ! queue ! nvv4l2decoder ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=960,height=540,format=NV12” ! queue ! comp2.sink_2 nvcompositor name=comp2 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=0 sink_1::ypos=270 sink_1::width=960 sink_1::height=540 sink_2::xpos=960 sink_2::ypos=270 sink_2::width=960 sink_2::height=540 ! nvvidconv ! nvoverlaysink liveadder name=mix ! queue ! audioconvert ! audioresample ! alsasink device=“hw:0,3” a1. ! queue ! mix.sink_0 a2. ! queue ! mix.sink_1 t1. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! videorate ! video/x-raw, width=1920, height=1080, framerate=1/2 ! jpegenc ! multifilesink location=rtsp1.jpg t2. ! h264parse ! queue ! nvv4l2decoder ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=960,height=540,format=NV12” ! multifilesink location=rtsp2.jpg