Hello everyone,
configuration :
jeston TX2 + pencv 3.4 + cuda 9.0
build information:
General configuration for OpenCV 3.4.1-dev =====================================
Version control: 3.4.1-9-gec0bb66-dirty
Extra modules:
Location (extra): /home/nvidia/opencv_contrib/modules
Version control (extra): 3.4.1
Platform:
Timestamp: 2019-03-22T15:04:56Z
Host: Linux 4.4.38-tegra aarch64
CMake: 3.5.1
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/make
Configuration: RELEASE
CPU/HW features:
Baseline: NEON FP16
required: NEON
disabled: VFPV3
C/C++:
Built as dynamic libs?: YES
C++ Compiler: /usr/bin/c++ (ver 5.4.0)
C++ flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -O3 -DNDEBUG -DNDEBUG
C++ flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -g -O0 -DDEBUG -D_DEBUG
C Compiler: /usr/bin/cc
C flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fopenmp -O3 -DNDEBUG -DNDEBUG
C flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fopenmp -g -O0 -DDEBUG -D_DEBUG
Linker flags (Release):
Linker flags (Debug):
ccache: NO
Precompiled headers: YES
Extra dependencies: dl m pthread rt /usr/lib/aarch64-linux-gnu/libGLU.so /usr/lib/aarch64-linux-gnu/libGL.so cudart nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cufft -L/usr/local/cuda-9.0/lib64
3rdparty dependencies: tbb
OpenCV modules:
To be built: aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv datasets dnn dnn_objdetect dpm face features2d flann freetype fuzzy hdf hfs highgui img_hash imgcodecs imgproc java_bindings_generator line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 python_bindings_generator reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab viz xfeatures2d ximgproc xobjdetect xphoto
Disabled: js world
Disabled by dependency: -
Unavailable: cnn_3dobj dnn_modern java matlab ovis sfm
Applications: tests perf_tests examples apps
Documentation: NO
Non-free algorithms: NO
GUI:
QT: YES (ver 5.5.1)
QT OpenGL support: YES (Qt5::OpenGL 5.5.1)
GTK+: NO
OpenGL support: YES (/usr/lib/aarch64-linux-gnu/libGLU.so /usr/lib/aarch64-linux-gnu/libGL.so)
VTK support: YES (ver 5.10.1)
Media I/O:
ZLib: /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.8)
JPEG: /usr/lib/aarch64-linux-gnu/libjpeg.so (ver )
WEBP: build (ver encoder: 0x020e)
PNG: /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.2.54)
TIFF: /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.6)
JPEG 2000: /usr/lib/aarch64-linux-gnu/libjasper.so (ver 1.900.1)
OpenEXR: build (ver 1.7.1)
Video I/O:
DC1394: NO
FFMPEG: YES
avcodec: YES (ver 56.60.100)
avformat: YES (ver 56.40.101)
avutil: YES (ver 54.31.100)
swscale: YES (ver 3.1.101)
avresample: NO
GStreamer:
base: YES (ver 1.8.3)
video: YES (ver 1.8.3)
app: YES (ver 1.8.3)
riff: YES (ver 1.8.3)
pbutils: YES (ver 1.8.3)
libv4l/libv4l2: 1.10.0 / 1.10.0
v4l/v4l2: linux/videodev2.h
gPhoto2: NO
Parallel framework: TBB (ver 2018.0 interface 10001)
Trace: YES (built-in)
Other third-party libraries:
Lapack: NO
Eigen: YES (ver 3.2.92)
Custom HAL: YES (carotene (ver 0.0.1))
Protobuf: build (3.5.1)
NVIDIA CUDA: YES (ver 9.0, CUFFT CUBLAS FAST_MATH)
NVIDIA GPU arch: 62
NVIDIA PTX archs:
OpenCL: YES (no extra features)
Include path: /home/nvidia/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load
Python 2:
Interpreter: /usr/bin/python2.7 (ver 2.7.12)
Libraries: /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.12)
numpy: /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.0)
packages path: lib/python2.7/dist-packages
Python 3:
Interpreter: /usr/bin/python3 (ver 3.5.2)
Libraries: /usr/lib/aarch64-linux-gnu/libpython3.5m.so (ver 3.5.2)
numpy: /usr/lib/python3/dist-packages/numpy/core/include (ver 1.11.0)
packages path: lib/python3.5/dist-packages
Python (for build): /usr/bin/python2.7
Java:
ant: NO
JNI: NO
Java wrappers: NO
Java tests: NO
Matlab: NO
Install to: /usr/local
-----------------------------------------------------------------
we are trying to capture a video stream from a titathink IP camera with the dimension 1280 * 720 @ 30 fps. on which is planned to do treatments with Opencv (cv :: equalizeHist, cv :: GaussianBlu, cv :: resize …and other treatments)
the camera can provide different Image compression methode H.264, MPEG-4, MJPEG
the ip address of the default camera is rtsp://192.168.1.111:80/live/0/h264.sdp but I find that we can use this address also rtsp://192.168.1.111:80/live/0/mjpeg.sdp
We try this code:
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <stdio.h>
#include <iostream>
using namespace std;
using namespace cv;
int main(int argc, char const *argv[])
{
int im_width=1200;
int im_height=720;
VideoCapture cap1("rtsp://192.168.1.111:80/live/0/h264.sdp");
Mat img1, img_res1;
while (1) {
cap1 >> img1;
resize(img1, img_res1, Size(im_width, im_height));
imshow("IMG1", img_res1);
waitKey(1);
}
return 0;
}
I know that I have a latency 5 seconds so I use the second address with the same code to reduce latency
.....
VideoCapture cap1("rtsp://192.168.1.111:80/live/0/mjpeg.sdp");
.....
and after the addition of the opencv processing it has become clear that the best solution is to use the decoding of the frames via the GPU directly.
after trying to install ffmeg on the jetson TX2 , we found that the ffmpeg library is only supported by desktop gpu.
by loading on the forum, I found that we can use gstreamer accelerate via GPU
#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>
int main(void)
{
const char *gst = "rtspsrc location=rtsp://192.168.1.111:80/live/0/mjpeg.sdp latency=0 ! application/x-rtp, media=(string)video \
! decodebin ! video/x-raw, format=(string)NV12 \
! videoconvert ! video/x-raw, format=(string)BGR \
! appsink";
cv::VideoCapture cap(gst);
if( !cap.isOpened() )
{
std::cout << "Error: Cv::VideoCapture.open() failed" << std::endl;
return -1;
}
unsigned int width = cap.get(CV_CAP_PROP_FRAME_WIDTH);
unsigned int height = cap.get(CV_CAP_PROP_FRAME_HEIGHT);
unsigned int pixels = width*height;
std::cout <<"Frame size : "<<width<<" x "<<height<<", "<<pixels<<" Pixels "<<std::endl;
cv::namedWindow("RTSP_Preview", CV_WINDOW_AUTOSIZE);
cv::Mat frame_in(width, height, CV_8UC3);
for(;;)
{
if (!cap.read(frame_in)) {
std::cout<<"Capture read error"<<std::endl;
break;
}
cv::imshow("RTSP_Preview",frame_in);
cv::waitKey(1); // let imshow draw
}
cap.release();
return 0;
}
compile with this command :
gcc -std=c++11 `pkg-config --cflags opencv` `pkg-config --libs opencv` test_gst.cpp -o gstreamer_view -lstdc++ -lopencv_core -lopencv_highgui -lopencv_videoio
when we tested this code with the links rtsp://192.168.1.111:80/live/0/mjpeg.sdp
it shows me this error despite this links works with cv::videocapture !!!
./gstreamer_view VIDEOIO ERROR: V4L: device rtspsrc location=rtsp://192.168.1.111:80/live/0/mjpeg.sdp latency=0 ! application/x-rtp, media=(string)video ! decodebin ! video/x-raw, format=(string)NV12 ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels
OpenCV(3.4.1-dev) Error: Unspecified error (GStreamer: unable to start pipeline
) in cvCaptureFromCAM_GStreamer, file /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp, line 890
VIDEOIO(cvCreateCapture_GStreamer (CV_CAP_GSTREAMER_FILE, filename)): raised OpenCV exception:
OpenCV(3.4.1-dev) /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp:890: error: (-2) GStreamer: unable to start pipeline
in function cvCaptureFromCAM_GStreamer
Error: Cv::VideoCapture.open() failed
and trying with the second links that rtsp://192.168.1.111:80/live/0/h264.sdp
the example works …
./gstreamer_view
VIDEOIO ERROR: V4L: device rtspsrc location=rtsp://192.168.1.111:80/live/0/h264.sdp latency=0 ! application/x-rtp, media=(string)video ! decodebin ! video/x-raw, format=(string)NV12 ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels
NvMMLiteOpen : Block : BlockType = 261
TVMR: NvMMLiteTVMRDecBlockOpen: 7647: NvMMLiteBlockOpen
NvMMLiteBlockCreate : Block : BlockType = 261
TVMR: cbBeginSequence: 1179: BeginSequence 1280x720, bVPR = 0
TVMR: LowCorner Frequency = 100000
TVMR: cbBeginSequence: 1529: DecodeBuffers = 10, pnvsi->eCodec = 4, codec = 0
TVMR: cbBeginSequence: 1600: Display Resolution : (1280x720)
TVMR: cbBeginSequence: 1601: Display Aspect Ratio : (1280x720)
TVMR: cbBeginSequence: 1669: ColorFormat : 5
TVMR: cbBeginSequence:1683 ColorSpace = NvColorSpace_YCbCr601
TVMR: cbBeginSequence: 1809: SurfaceLayout = 3
TVMR: cbBeginSequence: 1902: NumOfSurfaces = 17, InteraceStream = 0, InterlaceEnabled = 0, bSecure = 0, MVC = 0 Semiplanar = 1, bReinit = 1, BitDepthForSurface = 8 LumaBitDepth = 8, ChromaBitDepth = 8, ChromaFormat = 5
TVMR: cbBeginSequence: 1904: BeginSequence ColorPrimaries = 2, TransferCharacteristics = 2, MatrixCoefficients = 2
Allocating new output: 1280x720 (x 17), ThumbnailMode = 0
OPENMAX: HandleNewStreamFormat: 3464: Send OMX_EventPortSettingsChanged : nFrameWidth = 1280, nFrameHeight = 720
Frame size : 1280 x 720, 921600 Pixels
TVMR: FrameRate = 39
TVMR: NVDEC LowCorner Freq = (130000 * 1024)
---> TVMR: Video-conferencing detected !!!!!!!!!
TVMR: FrameRate = 28.218775
TVMR: FrameRate = 28.141824
TVMR: FrameRate = 28.348046
TVMR: FrameRate = 28.237181
TVMR: FrameRate = 28.246912
^C
but running htop we found that we used 50% of the cpu and my problem is I have 3 seconds of latency
what is this problem here?? : —> TVMR: Video-conferencing detected !!!
is there a way to use gpu for reading the video stream of the IP camera without causing latency ??
thank you for reading so far