How to add audio to gstreamer pipeline for rtsp streaming

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Followed the thread of “VLC playing gstreamer flow” to compile the RTSP server on TX2
https://devtalk.nvidia.com/default/topic/1018689/jetson-tx2/vlc-playing-gstreamer-flow/post/5187270/

Was able to stream the video to a local VLC on TX2.

But need help to add audio to the pipeline.

Here are what worked so far. First to compile the test-launch as instructed.
Then to launch a rtsp server on tx2:

$ ./test-launch "( \
        nvcamerasrc sensor-id=0 \
        ! video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=I420 \
        ! nvvidconv flip-method=4 \
        ! video/x-raw, width=720, height=480, framerate=30/1, format=I420 \
        ! timeoverlay \
        ! omxh265enc \
        ! rtph265pay name=pay0 pt=96 \
        )"

The video can be viewed on VLC of TX2 with “network stream” URL as “rtsp://127.0.0.1:8554/test”. I couldn’t do “rtsp://:8554/test” on remote devices (laptop, android devices) yet, not even within the WiFi local network, which is a separate issue to be resolved.

A bigger problem is audio. This is a working audio pipeline (with DMIC on csi camera board):

gst-launch-1.0 nvcamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12’ ! nvvidconv flip-method=0 ! ‘video/x-raw, width=720, height=480, framerate=30/1, format=NV12’ ! omxh265enc ! mpegtsmux name=mux alsasrc ‘device=hw:tegrasndt186ref,1’ ! audioresample ! audio/x-raw,channels=2 ! queue ! voaacenc bitrate=32000 ! queue ! mux. mux. ! filesink location=1cam_mix_audio.mpg sync=true async=false qos=true

Then if I add the audio portion to the rtsp pipeline:

$ ./test-launch "( \
        nvcamerasrc sensor-id=0 \
        ! video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12 \
        ! nvvidconv flip-method=5 \
        ! video/x-raw, width=720, height=480, framerate=30/1, format=NV12 \
        ! omxh265enc \
        ! mpegtsmux name=mux alsasrc 'device=hw:tegrasndt186ref,1' \
        ! audioresample ! audio/x-raw,channels=2 ! queue ! voaacenc bitrate=32000 ! queue ! mux. mux.  \
        ! rtph265pay name=pay0 pt=96 \
        )"

Viewing the stream “rtsp://127.0.0.1:8554/test” on VLC would then get an error:

Your input can't be opened:
VLC is unable to open the MRL 'rtsp://127.0.0.1:8554/test'. 
Check the log for details.

I first thought the quotation mark in ‘device=hw:tegrasndt186ref,1’ could be the cause of the problem. But I tried \'device=hw:tegrasndt186ref,1\', or simply removed the quotation mark, neither worked.

Any idea?

2

We have verified ov5693 with USB audio input:

[USB Audio input]
$ arecord -l
(...skip)
card 2: M [Plantronics Blackwire C220 M], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0

$ gst-launch-1.0 nvcamerasrc num-buffers=300 ! omxh264enc ! queue ! mux. alsasrc num-buffers=1000 device="hw:2,0" ! voaacenc ! queue ! qtmux name=mux ! filesink location=a.mp4

Please refer to it and adapt to your case.

3

Thanks DaneLLL.

With the helps from @jonathanh, I got the camera + DMIC working using command such as
gst-launch-1.0 nvcamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12’ ! nvvidconv flip-method=0 ! ‘video/x-raw, width=720, height=480, framerate=30/1, format=NV12’ ! omxh265enc ! mpegtsmux name=mux alsasrc ‘device=hw:1,1’ ! voaacenc ! queue ! mux. mux. ! filesink location=1cam_mixer_audio.mp4

Now I’m trying to take one step further towards streaming, following your example of RTSP as in
https://devtalk.nvidia.com/default/topic/1018689/jetson-tx2/vlc-playing-gstreamer-flow/post/5187270/

But I ran into a problem of converting the gst command for the RTSP server. If I directly plugin the gst pipeline to the test-launch() method, for example (just replace filesink with rtph265pay):

$ ./test-launch “(
nvcamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12’ ! nvvidconv flip-method=0 ! ‘video/x-raw, width=720, height=480, framerate=30/1, format=NV12’ ! omxh265enc ! mpegtsmux name=mux alsasrc ‘device=hw:1,1’ ! voaacenc ! queue ! mux. mux.
! rtph265pay name=pay0 pt=96
)”

I’d get an error shown in the terminal:

(test-launch:22568): GStreamer-CRITICAL **: gst_element_make_from_uri: assertion ‘gst_uri_is_valid (uri)’ failed

If I removed the quotation marks in the command, there would be no error in the terminal. When I started VLC on a remote device under the same local WiFi network, however the VLC failed to open the input stream at “rtsp://127.0.0.1:8554/test”.

Was my “test-launch” command correct?

4

Please try

./test-launch "nvcamerasrc ! omxh264enc ! video/x-h264,stream-format=(string)avc ! rtph264pay name=pay0 pt=96 audiotestsrc is-live=0 ! audioconvert ! audio/x-raw,rate=(int)8000,channels=(int)1 ! alawenc ! rtppcmapay pt=97 name=pay1"
5

@DaneLLL, thanks for your direction.
Any idea about how to code the selection of ALSA device? In my case it is " alsasrc ‘device=hw:1,1’ "
I think the quotation mark " ’ " could be an issue. The quote is needed due to a comma in the device definition. However the quote might not be allowed in rtsp test-launch. I tried to protect it with " / ", or “///”, but no luck.

6

Please try test-launch “()”
I see it here How to stream MIC input by RTSP with gstreamer - Stack Overflow

If it still does not work, please go to http://gstreamer-devel.966125.n4.nabble.com/

7

Thanks, DaneLLL. I’m still doing a trial-and-error. Let me give you a status report, perhaps other members may chip in with their experience as well.

Follow your instruction, I first download and compile the rtsp-server by

sudo apt-get install libgstrtspserver-1.0 libgstreamer1.0-dev
gcc test-launch.c -o test-launch $(pkg-config --cflags --libs gstreamer-1.0 gstreamer-rtsp-server-1.0)

Then I ran the command to start the rtsp server (single camera + DMIC on a customized CSI board)
export GST_DEBUG=3
./test-launch “(nvcamerasrc sensor-id=0
! video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12 ! nvvidconv flip-method=0 ! video/x-raw, width=640, height=480, framerate=30/1, format=NV12
! omxh265enc ! mpegtsmux name=mux alsasrc “device=hw:1,1” ! audioresample ! audio/x-raw,rate=48000,channels=2 ! queue ! voaacenc bitrate=32000
! queue ! mux. mux.
!rtph265pay name=pay0 pt=96
)”

The screen prompts
stream ready at rtsp://127.0.0.1:8554/test

So far so good.

Then I start VLC on TX2 and set the network stream to “rtsp://127.0.0.1:8554/test”, and VLC failed to open the stream. On the rtsp-server terminal, I got following message:

0:00:20.021461402  3740       0x57bca0 WARN                     omx gstomx.c:2836:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /home/nvidia/.config:/etc/xdg/xdg-ubuntu:/usr/share/upstart/xdg:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)
0:00:20.053676967  3740       0x57bca0 ERROR           GST_PIPELINE grammar.y:642:gst_parse_perform_link: could not link mux to pay0
0:00:20.053763302  3740       0x57bca0 WARN        rtspmediafactory rtsp-media-factory.c:1184:default_create_element: recoverable parsing error: could not link mux to pay0

Available Sensor modes : 
2592 x 1944 FR=15.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1920 x 1080 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=60.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
0:00:23.265672256  3740   0x7f6c002ed0 FIXME                default gstutils.c:3766:gst_pad_create_stream_id_internal:<nvcamerasrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:23.266579158  3740   0x7f6c002f20 FIXME                default gstutils.c:3766:gst_pad_create_stream_id_internal:<alsasrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 1920x1080 FrameRate = 30.000000 ...

0:00:23.267433645  3740   0x7f6c023c50 FIXME                default gstutils.c:3766:gst_pad_create_stream_id_internal:<appsrc1:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:23.267594476  3740   0x7f6c023f70 FIXME                default gstutils.c:3766:gst_pad_create_stream_id_internal:<appsrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:23.278965012  3740   0x7f6c002f20 WARN                    alsa pcm_hw.c:1241:snd_pcm_hw_get_chmap: alsalib error: Cannot read Channel Map ctl
: No such file or directory
0:00:23.399411874  3740   0x7f6c002ed0 FIXME           videoencoder gstvideoencoder.c:606:gst_video_encoder_setcaps:<omxh265enc-omxh265enc0> GstVideoEncoder::reset() is deprecated
Framerate set to : 30 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 8 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
0:00:23.403335033  3740   0x7f6c002ed0 WARN             omxvideoenc gstomxvideoenc.c:1860:gst_omx_video_enc_set_format:<omxh265enc-omxh265enc0> Error setting temporal_tradeoff 0 : Vendor specific error (0x00000001)
===== NVENC blits (mode: 1) into block linear surfaces =====
0:00:23.661122146  3740   0x7f38002a30 WARN             omxvideoenc gstomxvideoenc.c:1383:gst_omx_video_enc_loop:<omxh265enc-omxh265enc0> error: Internal data stream error.
0:00:23.661182914  3740   0x7f38002a30 WARN             omxvideoenc gstomxvideoenc.c:1383:gst_omx_video_enc_loop:<omxh265enc-omxh265enc0> error: stream stopped, reason not-linked
0:00:23.661372288  3740   0x7f740828a0 WARN               rtspmedia rtsp-media.c:2164:default_handle_message: 0x7f740853c0: got error Internal data stream error. (/dvs/git/dirty/git-master_linux/external/gstreamer/gst-omx/omx/gstomxvideoenc.c(1383): gst_omx_video_enc_loop (): /GstPipeline:media-pipeline/GstBin:bin0/GstOMXH265Enc-omxh265enc:omxh265enc-omxh265enc0:
stream stopped, reason not-linked)
0:00:23.661466143  3740       0x57bca0 WARN               rtspmedia rtsp-media.c:2459:wait_preroll: failed to preroll pipeline
0:00:23.661510014  3740       0x57bca0 WARN               rtspmedia rtsp-media.c:2749:gst_rtsp_media_prepare: failed to preroll pipeline
0:00:24.775084260  3740       0x57bca0 ERROR             rtspclient rtsp-client.c:729:find_media: client 0x5fd0e0: can't prepare media
0:00:24.780824359  3740       0x57bca0 ERROR             rtspclient rtsp-client.c:2239:handle_describe_request: client 0x5fd0e0: no media

Is there something wrong in my GST pipeline?

By the way, nearly identical, the pipeline below work with UDP on the same TX2 machine
gst-launch-1.0 nvcamerasrc sensor-id=0
! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12’ ! nvvidconv flip-method=0 ! ‘video/x-raw, width=640, height=480, framerate=30/1, format=NV12’
! omxh265enc ! mpegtsmux name=mux alsasrc ‘device=hw:1,1’ ! audioresample ! audio/x-raw,rate=48000,channels=2 ! queue ! voaacenc bitrate=32000
! queue ! mux. mux.
! udpsink host=127.0.0.1 port=3001

To receive on VLC, “upd://@:3001”

8

Went back to run the basic color-bar test, I realized the color-bar works on localhost but not on remove LAN device.

streaming the color bar:

export GST_DEBUG=3
./test-launch "videotestsrc ! omxh265enc ! rtph265pay name=pay0 pt=96"

And assume my TX2’s LAN ip address is 192.168.1.142 (WiFi wlan0), run VLC on android phone with “rtsp://192.168.1.142:8554/test” would get following errors on the rtsp-server terminal on TX2,

0:00:18.455810284 17071       0x57bca0 ERROR             rtspclient rtsp-client.c:1239:handle_play_request: client 0x5fd0e0: no aggregate path /test/stream=0
0:00:18.459218715 17071       0x57bca0 ERROR             rtspclient rtsp-client.c:903:handle_teardown_request: client 0x5fd0e0: no aggregate path /test/stream=0