Trying to use detectnet on mjpeg source

konstantin3 · December 24, 2020, 7:12am

Hi,

I have a webcam that produces mjpeg output. I would like to use this camera with detectnet to recognize if a person is coming into an area where they are not supposed to be.
I managed to cobble together an gst-launch queue that reads and parses the source and i though of sending this to an rtp input. However running detectnet rtp://@:1234 does not seem to receive anything. Any clue what I might be doing wrong?

call for the sending gst-launch is:

gst-launch-1.0 -v souphttpsrc location=http://192.168.1.1/snapshot.cgi user-id=user user-pw=password do-timestamp=true ! multipartdemux ! image/jpeg,width=1280,height=960 ! jpegdec ! videoconvert ! x264enc tune=zerolatency bitrate=90000 speed-preset=superfast ! rtph264pay ! udpsink host=jetson port=1234
the sending host is not the jetson
since i have no display on the jetson i tried to forward detectnet’s output somewhere else via rtp, so i called detectnet with:
detectnet --input-codec=h264 rtp://@:1234 rtp://destinationhost:1234

however detectnet just gives me ‘failed to capture video frame’ messages but way to few to indicate it is receiving something. According to tcpdump packets are sent.

If there is a simpler way to get the data from the webcam please do tell :)

Kind Regards,

Konstantin

dusty_nv · December 22, 2020, 8:30pm

Hi @konstantin3, is your webcam connected over USB? If so, you can try running it directly with detectnet program (it supports mjpeg). You may want to try the video-viewer utility first.

If your webcam is not USB, does it natively support RTP/RTSP? If so, detectnet program can also do those (without needing the intermediary PC pipeline).

If not, I take it that your gst-launch pipeline on your PC is taking that camera’s cgi web interface, and converting it to RTP. Before you try viewing the RTP feed on your Jetson, are you able to view the RTP feed from your PC? It would be good to test that part of the pipeline is working first.

If you run ifconfig from your Jetson, can you see recieve (rx) network traffic increasing?

konstantin3 · December 23, 2020, 12:12am

No it is an IP-Cam sorry for the confusion.
It only offers the cgi mjpeg interface. And yes the GST-Pipe does work if i set up an audiovideosink i even used it and read on the jetson, let omxh264 reencode it and send it to the machine I am sittting in front of. Might it be, that I need to tune some bitrate parameters etc in sending to detecnet?

dusty_nv · December 23, 2020, 3:19pm

OK, gotcha. Can you try running this video-viewer command and post the console output?

video-viewer --input-codec=h264 rtp://@:1234 dump.mp4

Sometimes it can take some seconds for the RTP stream to start being received, but my guess is you have waited already.

konstantin3 · December 23, 2020, 4:06pm

I will paste it as it is not too long:

gstreamer] initialized gstreamer, version 1.14.5.0
[gstreamer] gstDecoder -- creating decoder for 127.0.0.1
ESC[0;33m[gstreamer] gstDecoder -- resource discovery not supported for RTP streams
ESC[0m[gstreamer] gstDecoder -- pipeline string:
[gstreamer] udpsrc port=1234 multicast-group=127.0.0.1 auto-multicast=true caps="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264" ! rtph264depay ! h264parse ! omxh264dec ! video/x-raw ! appsink name=mysink
ESC[0;32m[video]  created gstDecoder from rtp://@:1234
ESC[0m------------------------------------------------
gstDecoder video options:
------------------------------------------------
  -- URI: rtp://@:1234
     - protocol:  rtp
     - location:  127.0.0.1
     - port:      1234
  -- deviceType: ip
  -- ioType:     input
  -- codec:      h264
  -- width:      0
  -- height:     0
  -- frameRate:  0.000000
  -- bitRate:    0
  -- numBuffers: 4
  -- zeroCopy:   true
  -- flipMethod: none
  -- loop:       0
------------------------------------------------
ESC[0;33m[gstreamer] gstEncoder -- codec not specified, defaulting to H.264
ESC[0m[gstreamer] gstEncoder -- pipeline launch string:
[gstreamer] appsrc name=mysource is-live=true do-timestamp=true format=3 ! omxh264enc bitrate=4000000 ! video/x-h264 !  h264parse ! qtmux ! filesink location=dump.mp4
ESC[0;32m[video]  created gstEncoder from file:///home/elwood/dump.mp4
ESC[0m------------------------------------------------
gstEncoder video options:
------------------------------------------------
  -- URI: file:///home/elwood/dump.mp4
     - extension: mp4
  -- deviceType: file
  -- ioType:     output
  -- codec:      h264
  -- width:      0
  -- height:     0
  -- frameRate:  30.000000
  -- bitRate:    4000000
  -- numBuffers: 4
  -- zeroCopy:   true
  -- flipMethod: none
  -- loop:       0
------------------------------------------------
ESC[0;31m[OpenGL] failed to open X11 server connection.
ESC[0mESC[0;31m[OpenGL] failed to create X11 Window.
ESC[0m[gstreamer] opening gstDecoder for streaming, transitioning pipeline to GST_STATE_PLAYING
[gstreamer] gstreamer changed state from NULL to READY ==> mysink
[gstreamer] gstreamer changed state from NULL to READY ==> capsfilter0
[gstreamer] gstreamer changed state from NULL to READY ==> omxh264dec-omxh264dec0
[gstreamer] gstreamer changed state from NULL to READY ==> h264parse0
[gstreamer] gstreamer changed state from NULL to READY ==> rtph264depay0
[gstreamer] gstreamer changed state from NULL to READY ==> udpsrc0
[gstreamer] gstreamer changed state from NULL to READY ==> pipeline0
[gstreamer] gstreamer changed state from READY to PAUSED ==> capsfilter0
[gstreamer] gstreamer changed state from READY to PAUSED ==> omxh264dec-omxh264dec0
[gstreamer] gstreamer changed state from READY to PAUSED ==> h264parse0
[gstreamer] gstreamer changed state from READY to PAUSED ==> rtph264depay0
[gstreamer] gstreamer stream status CREATE ==> src
[gstreamer] gstreamer changed state from READY to PAUSED ==> udpsrc0
[gstreamer] gstreamer changed state from READY to PAUSED ==> pipeline0
[gstreamer] gstreamer message new-clock ==> pipeline0
[gstreamer] gstreamer stream status ENTER ==> src
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> omxh264dec-omxh264dec0
gstreamer] gstreamer changed state from PAUSED to PLAYING ==> h264parse0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> rtph264depay0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> udpsrc0
[gstreamer] gstreamer message stream-start ==> pipeline0
ESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame
ESC[0mESC[0;31mvideo-viewer:  failed to capture video frame

the ‘failed to capture’ are continuing

dusty_nv · December 23, 2020, 4:08pm

OK thanks - can you try running this other test pipeline from your PC, and check to see if your Jetson can recieve/display it ok through the video-viewer app again?

# run from PC (after disabling your other PC pipeline)
$ gst-launch-1.0 -v videotestsrc ! video/x-raw,width=300,height=300,framerate=30/1 ! x264enc ! rtph264pay ! udpsink host=jetson port=1234

If that test pipeline works, I would try changing a couple things in your camera pipeline to see if they make a difference:

Change the resolution from 1280x960 to 1280x720
Remove tune=zerolatency bitrate=90000 speed-preset=superfast

konstantin3 · December 23, 2020, 5:27pm

How shall I receive on the jetson videoviewer? If yes then not getting better.
I also tried to send the stream to a gst-udpsrc receiver on the same host also nothing.
I have another sender that kind of works:
gst-launch-1.0 -v ximagesrc ! video/x-raw,framerate=20/1 ! videoscale ! videoconvert ! x264enc tune=zerolatency bitrate=500 speed-preset=superfast ! rtph264pay ! udpsink host=localhost port=1234

receives with gst-launch-1.0 -vvv udpsrc port=1234 caps = “application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96” ! rtph264depay ! decodebin ! videoconvert ! autovideosink

video-viewer ignores it

I do think playing with GST_DEBUG that the reason is somewhere in the rtph264 repay module.

dusty_nv · December 23, 2020, 5:32pm

Maybe try changing to port 5000 in case there is some system permission/access issue on either end with port 1234.

Try with this simpler pipeline from your PC first, and then video-viewer to view it:

# from your PC
gst-launch-1.0 -v videotestsrc ! video/x-raw,width=300,height=300,framerate=30/1 ! x264enc ! rtph264pay ! udpsink host=jetson port=5000

# from your Jetson
$ video-viewer --input-codec=h264 rtp://@:5000

konstantin3 · December 23, 2020, 6:36pm

Same Result no reaction on the jetson side. In the final versiony the jetson should actually not be the receiver via rtp. The jetson can collect the data via gst from the ip-cam itself. The question is: what is the best way to get this into detectnet (preferably from python since in the end I want to create a script that if detectnet discovers a person, sends an email).

Regards,

Konstantin

dusty_nv · December 23, 2020, 7:23pm

To use a custom GStreamer pipeline such as yours, it might be easiest to use cv2.VideoCapture interface with your GStreamer pipeline, and then convert the OpenCV image to use with detectnet.py like in this sample that uses the jetson.utils.cudaFromNumpy() function:

https://github.com/dusty-nv/jetson-utils/blob/833fc7998e34d852672277730a11aeed90024959/python/examples/cuda-from-cv.py

The first step would be to test that cv2.VideoCapture can get frames from your custom GStreamer pipeline from your IP camera.

konstantin3 · December 23, 2020, 11:47pm

how do I put in the custom gstreamer pipeline in this example. Is there a python equivalent of the commmandline call I am doing?

konstantin3 · December 24, 2020, 10:57am

Hi,

I managed to get it working in the way i was thinking it should. The trick was to add:
! ‘video/x-h264, stream-format=(string)byte-stream’ !
after the omxh264enc.
Now I have a final issue: the quality is very low (the source does deliver higher resolution) how do I tell omxh264enc to produce better quality pictures? I tried converting other videos using hw-accellerated gstreamer as well and either there was nearly no compression or the quality was serverely downgraded. Compared to converting video sources with ffmpeg I am wondering what are the missing parameters.

However it would be great if I can use the gstreamer queue directly as input in my python script instead of running two processes.

dusty_nv · January 6, 2021, 2:26am

Hi @konstantin3, you can set the bitrate attribute of omxh264enc element (for more info, run gst-inspect-1.0 omxh264enc)

 control-rate        : Bitrate control method
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstOMXVideoEncControlRate" Default: 1, "variable"
                           (0): disable          - Disable
                           (1): variable         - Variable
                           (2): constant         - Constant
                           (3): variable-skip-frames - Variable Skip Frames
                           (4): constant-skip-frames - Constant Skip Frames
  bitrate             : Target bitrate
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4000000
  peak-bitrate        : Peak bitrate in variable control-rate
                         The value must be >= bitrate
                         (1.2*bitrate) is set by default(Default: 0)
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0

I believe the default is 4mbps VBR.