Jetson Nano mix 2 USB audio sources to a video

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

Hi,

I am using the below command to record and preview 2 1080p usb video sources one video with audio. Below is the pipeline

gst-launch-1.0 -e v4l2src device=/dev/video1 ! tee name=t1 t1. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 profile=4 ! queue ! h264parse ! queue ! mux. pulsesrc device=“alsa_input.usb-VXIS_Inc_ezcap_U3_capture-02.analog-stereo” ! queue ! audio/x-raw,width=16,depth=16,rate=44100,channel=1 ! audioconvert ! voaacenc ! aacparse ! mp4mux name=mux ! filesink location=feed1.mp4 t1. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=503,height=250,framerate=30/1,format=NV12” ! queue ! nvoverlaysink overlay-x=0 overlay-y=50 overlay-w=503 overlay-h=250 overlay=1 v4l2src device=/dev/video0 io-mode=2 do-timestamp=true ! tee name=t2 t2. ! queue ! image/jpeg,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw,width=1920,height=1080,framerate=30/1 ! nvvidconv ! queue ! ‘video/x-raw(memory:NVMM),framerate=30/1,format=NV12’ ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 profile=4 ! queue ! h264parse ! queue ! mp4mux ! filesink location=feed2.mp4 t2. ! queue ! image/jpeg,width=1920,height=1080,framerate=30/1 ! nvjpegdec ! video/x-raw,framerate=30/1 ! nvvidconv ! queue ! ‘video/x-raw(memory:NVMM),framerate=30/1,format=NV12’ ! queue ! nvoverlaysink overlay-x=504 overlay-y=50 overlay-w=504 overlay-h=250 overlay=2

I need to mix another USB audio source to the first file with the current USB audio source. Which means 2 USB audio sources has to be mixed with the first video file.

How can I achieve this scenario?

Thank You

2

If you want to have 2 separate audio tracks, you would do something like:

gst-launch-1.0 -e mp4mux name=mux ! filesink location=test.mp4     videotestsrc is-live=true ! nvvidconv ! nvv4l2h264enc ! h264parse ! queue ! mux.video_0     audiotestsrc wave=sine ! voaacenc ! queue ! mux.audio_0     audiotestsrc wave=ticks ! voaacenc ! queue ! mux.audio_1

Or if you just want to mix both audio sources into a single track:

gst-launch-1.0 -e mp4mux name=mux ! filesink location=test.mp4     videotestsrc is-live=true ! nvvidconv ! nvv4l2h264enc ! h264parse ! queue ! mux.video_0     audiomixer name=mixer ! voaacenc ! queue ! mux.audio_0     audiotestsrc wave=sine ! queue ! mixer.sink_0     audiotestsrc wave=ticks ! queue ! mixer.sink_1
3

Thank you very much for the help @Honey_Patouceul. I was able mix the 2 audios into one single track

4

Hi @Honey_Patouceul

I have an issue after the 2 audios are mixed. The mixed audio contains artifacts like click sounds is there a method to solve this issue

Thanks in advance

5

If you are still using audiotestrc wave=ticks then it is normal as I used this for distinguishing from continuous sine wave for checking mixer.
If not, this is unexpected but I can’t tell more.

6

Hi @Honey_Patouceul,

Thank you for the reply. I am using two USB audio sources to mix as shown below which gives those sounds. Single Audio sources are working fine separately. I think I have used the audiomixer properly.

gst-launch-1.0 -e mp4mux name=mux ! filesink location=feed1mp1080HS.mp4 v4l2src device=/dev/video1 ! tee name=t1 t1. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 profile=4 ! queue ! h264parse ! queue ! mux.video_0 audiomixer name=mixer ! audioconvert ! voaacenc ! aacparse ! mux.audio_0 pulsesrc device=“alsa_input.usb-VXIS_Inc_ezcap_U3_capture-02.analog-stereo” ! queue ! audio/x-raw,width=16,depth=16,rate=44100,channel=1 ! queue ! mixer.sink_0 pulsesrc device=“alsa_input.usb-C-Media_Electronics_Inc._USB_Advanced_Audio_Device-00.analog-stereo” ! queue ! audio/x-raw,width=16,depth=16,rate=44100,channel=1 ! queue ! mixer.sink_1 t1. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=959,height=540,framerate=30/1,format=NV12” ! queue ! nvoverlaysink overlay-x=0 overlay-y=270 overlay-w=959 overlay-h=540 overlay=1 v4l2src device=/dev/video2 ! tee name=t2 t2. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 profile=4 ! queue ! h264parse ! queue ! mp4mux ! filesink location=feed2mp1080HS.mp4 t2. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=960,height=540,framerate=30/1,format=NV12” ! queue ! nvoverlaysink overlay-x=960 overlay-y=270 overlay-w=960 overlay-h=540 overlay=2

7

Hi @Honey_Patouceul ,

I can run the below gstreamer command successfully without any errors to save 2 mp4 files

gst-launch-1.0 -e mp4mux name=mux ! filesink location=feed1mp1080HS.mp4 v4l2src device=/dev/video1 ! tee name=t1 t1. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 profile=4 ! queue ! h264parse ! queue ! mux.video_0 audiomixer name=mixer ! audioconvert ! voaacenc ! aacparse ! mux.audio_0 pulsesrc device=“alsa_input.usb-VXIS_Inc_ezcap_U3_capture-02.analog-stereo” ! queue ! audio/x-raw,width=16,depth=16,rate=44100,channel=1 ! queue ! mixer.sink_0 pulsesrc device=“alsa_input.usb-C-Media_Electronics_Inc._USB_Advanced_Audio_Device-00.analog-stereo” ! queue ! audio/x-raw,width=16,depth=16,rate=44100,channel=1 ! queue ! mixer.sink_1 t1. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=959,height=540,framerate=30/1,format=NV12” ! queue ! nvoverlaysink overlay-x=0 overlay-y=270 overlay-w=959 overlay-h=540 overlay=1 v4l2src device=/dev/video2 ! tee name=t2 t2. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 profile=4 ! queue ! h264parse ! queue ! mp4mux ! filesink location=feed2mp1080HS.mp4 t2. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=960,height=540,framerate=30/1,format=NV12” ! queue ! nvoverlaysink overlay-x=960 overlay-y=270 overlay-w=960 overlay-h=540 overlay=2

I need to save 2 .ts files and I tried the below command but it gives an error stating
Warning: erroneous pipeline: could not link queue3 to mux

gst-launch-1.0 -e mpegtsmux name=mux ! filesink location=feed1mp1080HSTS.ts v4l2src device=/dev/video1 ! tee name=t1 t1. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 profile=4 ! queue ! h264parse ! queue ! mux.video_0 audiomixer name=mixer ! queue ! audioconvert ! voaacenc ! aacparse ! mux.audio_0 pulsesrc device=“alsa_input.usb-VXIS_Inc_ezcap_U3_capture-02.analog-stereo” ! queue ! audio/x-raw,width=16,depth=16,rate=44100,channel=1 ! queue ! mixer.sink_0 pulsesrc device=“alsa_input.usb-C-Media_Electronics_Inc._USB_Advanced_Audio_Device-00.analog-stereo” ! queue ! audio/x-raw,width=16,depth=16,rate=44100,channel=1 ! queue ! mixer.sink_1 t1. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=959,height=540,framerate=30/1,format=NV12” ! queue ! nvoverlaysink overlay-x=0 overlay-y=270 overlay-w=959 overlay-h=540 overlay=1 v4l2src device=/dev/video2 ! tee name=t2 t2. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! nvv4l2h264enc maxperf-enable=1 bitrate=4000000 profile=4 ! queue ! h264parse ! queue ! mpegtsmux ! filesink location=feed2mp1080HSTS.ts t2. ! queue ! video/x-raw, width=1920, height=1080, framerate=30/1 ! nvvidconv ! queue ! “video/x-raw(memory:NVMM),width=960,height=540,framerate=30/1,format=NV12” ! queue ! nvoverlaysink overlay-x=960 overlay-y=270 overlay-w=960 overlay-h=540 overlay=2

How can I resolve this?

Thank You

8

If you look to plugin:

gst-inspect-1.0 mpegtsmux
...
  SINK template: 'sink_%d'
...

As you can see, its pads are just sink numbers, no matter it is video or audio or else.
So just try to change any mux.video_* and mux.audio_* to a common mux.sink_ numbering.

1 Like
9

Thank you @Honey_Patouceul