Kangalow, thank you very, very much for your effort. It is a definite improvement over what I had. Thanks for posting the video, too.
CPU usage seems to be sitting around 300% (presumably, apx. 75% of each of the four CPUs). I’m guessing that the jpegdec is to blame. Nvidia says that nvjpegdec is buggy (fixed in next release), but I have not heard the date for the next release.
I also got some feedback on the nv_omx_videomixer in that it is not really a mixer like videomixer is; perhaps it is poorly named(?).
Hi hilndr44,
The new release is out! LT4 21.1:
I have a little video of an install:
and some notes, but it’s pretty straightforward.
nvjpegdec appears to work in the new release, but my first experience with trying to use a webcam with gstreamer did not work. Once you have the new release installed, you should be able to substitute the jpegdec element with nvjpegdec. It will be interesting to see how much of a speedup occurs, and how much of the CPU resources are freed up.
Kangalow, I upgraded to R21.1, and swapped the jpegdec with nvjpegdec. It made no difference to CPU usage. Both are sitting at a CPU usage of between 290-320%. This either means that jpegdec is only a very minor part of the CPU usage, or nvjpegdec is not using hardware acceleration at all. :(
This test-case is with four 1280x720 mjpeg Axis M1104 IP cams attached to the same local switch that has the Jetson attached.
Hmm. I thought this is the part where nvjpegdec swoops down and saves the day! There are only a few parts here, decoding the image, compositing the image and then displaying it on the screen. If nvjpegdec actually does do hardware decoding, I would have thought you would have seen at least some reduction in CPU usage. As I recall, it’s some sort of DCT encoding, so while it’s not really complicated the hardware should have sped things up if only for having more resources concurrently working.
I couldn’t get rpaliwal_nvidia nv_omx_hdmi_videosink solution to work with 19.3, it might work better with 21.1. The advantage of that solution is that it does the compositing and display in one operation.
Which video sink are you using? It might be useful to put up your full pipeline again if it’s close to working.
Even with the CPU usage issue, is the current results that you are seeing anywhere close to acceptable or near video stream frame rate?
The actual results are “close” to acceptable. The framerate is having a little trouble keeping up, and I can see that it is dropping frames.
The pipeline I tested it with was the same as you have posted. I tried it with both the xvimagesink and nv_omx_hdmi_videosink; no difference, other than the size of the display window.
I couldn’t get simultaneous nv_omx_hdmi_videosink to work in R21.1 either.
The following single view pipeline takes apx. 58% of the CPU :
#!/bin/sh
gst-launch-0.10 -e -v
souphttpsrc location=“http://root:<PASSWD>@192.168.0.112/axis-cgi/mjpg/video.cgi?resolution=1280x720” timeout=5 do-timestamp=true is-live=true
! jpegparse
! nvjpegdec
! nv_omx_hdmi_videosink overlay-x=0 overlay-y=540 overlay-w=960 overlay-h=540 sync=false
I am trying to get the Jetson to display h264 from the Axis cameras, but I am having trouble coming up with a pipeline that will work.
Add to that the fact that videomixer will not accept NV12 (which is the output from nv_omx_h264dec), and I am stuck with the same problem of how to display multiple streams. But, one problem at a time… :)
The h264 pipeline below gives me this error :
ERROR: from element /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0/GstUDPSrc:udpsrc0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2625): gst_base_src_loop (): /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0/GstUDPSrc:udpsrc0:
streaming task paused, reason not-linked (-1)
gst-launch-0.10 -e -v
rtspsrc location=rtsp://root:@192.168.0.113:554/axis-media/media.amp?videocodec=h264&resolution=1280x720
! application/x-rtp,payload=96,encoding-name=H264
! rtph264depay
! nv_omx_h264dec
! nv_omx_hdmi_videosink overlay-x=0 overlay-y=540 overlay-w=960 overlay-h=540 sync=false
The nv_omx_h264 issue could be that you need a h264parse element beforehand.
! h264parse ! nv_omx_h264dec
I would also try with a queue element beforehand:
! h264parse ! queue ! nv_omxh264dec
The H264 decoder is definitely hardware, the CPU cores would have a hard time keeping up.
Unfortunately I don’t have access to an Axis, so I’m running out of ideas of what to try.
I’ll tell you what, I’m finding it difficult to be a fan of Gstreamer right now.
I decided to try gstreamer 1.0 (instead of 0.10, in which I have had no luck), just to see if I could get something to display.
I was partly successful (pipeline below); I get less than half a second of motion, but then the picture freezes.
gst-launch-1.0 -e -v
rtspsrc location=rtsp://root:@192.168.0.113:554/axis-media/media.amp?videocodec=h264
! application/x-rtp,width=1280,height=720,framerate=30/1
! rtph264depay
! h264parse
! video/x-h264,width=1280,height=720,framerate=30/1
! queue
! omxh264dec
! video/x-raw,width=1280,height=720,framerate=30/1
! nvhdmioverlaysink sync=false
That was my experience also. Need to file a bug report with NVIDIA.
On another note,
h264parse ! queue ! omxh264dec
did work correctly under 19.3. If you are able to get h264 out of your camera, you don’t need to use nvjpegdec which was the main reason to upgrade to LT4 21.1. This is some puzzle.
Also, nvjpegdec seems to produce poorer quality video than jpegdec.
Maybe it is just me, but it seems more grainy and less crisp.
I would really like to see omxh264dec working, but it would open up two more cans of worms : 1) videomixer will not handle the output of h264dec, and 2) without videomixer, I still cannot run simultaneous pipelines displayed over HDMI.
For now, I will have to be content with mjpeg, 320% CPU usage, dropped frames, and no recording capability.
I bought 5 of these puppies (four, plus a spare), so at least I can use one for debugging while the other 4 are in constant use (starting in mid-December). Hopefully, some answers/updates/patches are forthcoming.
OK, I now have a working solution with h264!
I was finding that the output of nv_omx_h264dec was not compatible with nv_omx_hdmi_videosink, so I had to insert an nvvidconv. However, the only common ground between nv_omx_h264dec and nvvidconv is video/x-nv-yuv in NV12 format (despite what gst-inspect claims!).
Then I had to place a caps filter on the output of nvvidconv to send I420 format to videomixer (the only format it seems to like from nvvidconv). I also figured that I might as well let nvvidconv down-scale it while it converted the formats.
I also needed the caps filter after rtph264depay. Note that if I removed any of the caps filters, then the pipeline would give me the dreaded (and non-descript) “Internal data flow error”.
I still have to use videobox, because I haven’t figured out how to get videomixer working correctly to display using xpos/ypos on a per-sink basis (whatever is chosen as stream #0 will take up the whole screen). I am hoping that this will help reduce the CPU usage. I could use some help with that part, please.
The result is really, really smooth, and with low latency (much better than the mjpeg solution).
The downside is that it still takes 295% of the CPU power.
Just one discrete stream takes 55%, so I am guessing that the combination of videobox(s) and videomixer is taking 295-(4x55)=75% of the CPU.
Pipeline is as follows :
#!/bin/sh
H264_DEPAY=“rtph264depay ! video/x-h264,width=1280,height=720,framerate=30/1”
H264_DECODE=“nv_omx_h264dec ! video/x-nv-yuv,format=(fourcc)NV12,framerate=30/1,width=1280,height=720”
H264_CONVERT=“nvvidconv ! video/x-raw-yuv,format=(fourcc)I420,framerate=30/1,width=960,height=540”
gst-launch-0.10 -e -v
rtspsrc location=rtsp://root:@192.168.0.121:554/axis-media/media.amp?videocodec=h264
! $H264_DEPAY
! $H264_DECODE
! $H264_CONVERT
! videobox left=-960 top=-540 border-alpha=0
! queue
! vmix.sink_4
rtspsrc location=rtsp://root:@192.168.0.122:554/axis-media/media.amp?videocodec=h264
! $H264_DEPAY
! $H264_DECODE
! $H264_CONVERT
! videobox top=-540 border-alpha=0
! queue
! vmix.sink_3
rtspsrc location=rtsp://root:@192.168.0.123:554/axis-media/media.amp?videocodec=h264
! $H264_DEPAY
! $H264_DECODE
! $H264_CONVERT
! videobox left=-960 border-alpha=0
! queue
! vmix.sink_2
rtspsrc location=rtsp://root:@192.168.0.124:554/axis-media/media.amp?videocodec=h264
! $H264_DEPAY
! $H264_DECODE
! $H264_CONVERT
! queue
! vmix.sink_1
videomixer name=vmix
! nv_omx_hdmi_videosink sync=false
Congratulations on getting it to work!
I guess another factor in trying to figure out the CPU load is how much work is it for an individual core to move one of the camera streams from the ethernet buffers to an appropriate place(s) in main memory. After all, it needs to get the bits, get them to the decoder, then back after they are decoded. I’d guess that a lot of this is done intelligently through DMA buffers, but the bits are the bits and eventually they have to move somewhere (hopefully to the screen) to get the next batch. Also, an idle system with nothing going on appears to use half of one core.
Do you have the CPU set to performance?
How do I set the CPU to performance?
I agree that the data transfer must take up some of the resources.
Do you know if nvvidconv is run on the GPU or in S/W?
From the Jetson Wiki:
http://elinux.org/Jetson/Performance
Maximizing CPU performance
To obtain full performance on the CPU (eg: for performance measurements or benchmarking or when you don't care about power draw), you can disable CPU scaling and force the 4 main CPU cores to always run at max performance until reboot:
echo 0 > /sys/devices/system/cpu/cpuquiet/tegra_cpuquiet/enable
echo 1 > /sys/devices/system/cpu/cpu0/online
echo 1 > /sys/devices/system/cpu/cpu1/online
echo 1 > /sys/devices/system/cpu/cpu2/online
echo 1 > /sys/devices/system/cpu/cpu3/online
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
You can place the code into /etc/rc.local to have it executed on Jetson startup.
I don’t know about nvvidconv.
I noticed that L4T 21.1 has a CPU/GPU profiler. I haven’t played with it yet, so I don’t know if it is applicable in this situation.