Torch/ Lua - Accessing onboard camera using OpenCV 3.1

I am trying to build an inference/ prediction model for a CNN using Torch/ Lua. I need to access the video feed from the onboard camera of Jetson TX1 using OpenCV 3.1. I have installed the torch-opencv (https://github.com/VisionLabs/torch-opencv) package for this purpose.

This link provides one of the demo programs in the package: https://github.com/VisionLabs/torch-opencv/blob/master/demo/videocap.lua. When I try to run it, I get the error:

Failed to open the default camera

I tried accessing the camera from the terminal using:

gst-launch-1.0 nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)640, height=(int)480, format=(string)I420, framerate=(fraction)30/1' ! nvtee ! nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e

and it worked (However, the window size did not match the resolution).

So, I read a few other threads (https://devtalk.nvidia.com/default/topic/904949/how-to-get-tx1-camera-in-opencv/, https://devtalk.nvidia.com/default/topic/925923/jetson-tx1/can-not-open-the-camera-on-tx1-via-opencv/post/4841289/#4841289) and tried the following, to open this camera using OpenCV 3.1, inside the above mentioned demo program (The -- indicates lines commented out):

--gst = "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280," ..
      " height=(int)720,format=(string)I420, framerate=(fraction)24/1 !" ..
      " nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert" ..
      " ! video/x-raw, format=(string)BGR ! appsink"
--local cap = cv.VideoCapture{gst}

local cap = cv.VideoCapture{"nvcamerasrc fpsRange=\"30.0 30.0\" ! 'video/x-raw(memory:NVMM)," ..
							" width=(int)640, height=(int)480, format=(string)I420, framerate=(fraction)30/1' !" ..
							" nvtee ! nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' !" ..
							" nvoverlaysink -e"}
--local cap = cv.VideoCapture{device=0}

However, the error persists. Is there any way to fix this issue? Has anyone accessed this camera using Torch/ Lua? Thanks in advance.

Hi,

Thanks for your question.
Could you check if you can open onboard camera via opencv first?

That’s what I have been trying to do. I have edited the question slightly to make things more clear. The torch-opencv package uses OpenCV 3.1 to open the camera. The demo program is an attempt to do that (among other things). I tried the following lines inside the demo program to open the camera:

--gst = "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280," ..
      " height=(int)720,format=(string)I420, framerate=(fraction)24/1 !" ..
      " nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert" ..
      " ! video/x-raw, format=(string)BGR ! appsink"
--local cap = cv.VideoCapture{gst}

local cap = cv.VideoCapture{"nvcamerasrc fpsRange=\"30.0 30.0\" ! 'video/x-raw(memory:NVMM)," ..
							" width=(int)640, height=(int)480, format=(string)I420, framerate=(fraction)30/1' !" ..
							" nvtee ! nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' !" ..
							" nvoverlaysink -e"}
--local cap = cv.VideoCapture{device=0}

Hi,

Thanks for your feedback.

Could you try if you can open onboard camera via this sample code?

#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>

int main(void)
{
    cv::VideoCapture cap("nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)24/1 ! nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink");

    if( !cap.isOpened() )
    {
        std::cout << "Not good, open camera failed" << std::endl;
        return 0;
    }

    cv::Mat frame;
    while(true)
    {
        cap >> frame;
        cv::imshow("Frame", frame);
        cv::waitKey(1);
    }
    return 0;
}
nvcc topic_1005835.cpp -lopencv_core -lopencv_highgui -lopencv_videoio -o test && ./test

Please make sure you have built opencv with gstreamer enabled.
For example,

cmake -D WITH_CUDA=ON -D CUDA_ARCH_BIN="5.3" -D CUDA_ARCH_PTX="" -D WITH_GSTREAMER=ON -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_EXAMPLES=OFF -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local ..

Please also check cmake report has gstreamer enabled

--   Video I/O:
......
--     GStreamer:                   
--       base:                      YES (ver 1.8.3)
--       video:                     YES (ver 1.8.3)
--       app:                       YES (ver 1.8.3)
--       riff:                      YES (ver 1.8.3)
--       pbutils:                   YES (ver 1.8.3)

I will try to install torch-opencv and update information to you later.
Thanks.

Thank you for the reply. I had not specifically enabled gstreamer while building OpenCV. I shall do that and update you. I shall also try the c++ code (I am busy for a couple of days with school work. Apologies for the possible delay in update).

I deleted the existing build folder inside opencv folder and created a new one. I ran the given CMAKE
command and checked the result. It shows ‘NO’ against gstreamer. I am using L4T Sample File 23.2 and corresponding jetpack.
.

Try installing gstreamer dev packages and re-run cmake.

sudo apt-get -y install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev

Thank You for the reply. That fixed the problem with CMAKE.

I executed the CMAKE command given and then tried the demo program with the following two lines:

gst = "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280," ..
      " height=(int)720,format=(string)I420, framerate=(fraction)24/1 !" ..
      " nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert" ..
      " ! video/x-raw, format=(string)BGR ! appsink"
local cap = cv.VideoCapture{gst}

instead of

local cap = cv.VideoCapture{"nvcamerasrc fpsRange=\"30.0 30.0\" ! 'video/x-raw(memory:NVMM)," ..
							" width=(int)640, height=(int)480, format=(string)I420, framerate=(fraction)30/1' !" ..
							" nvtee ! nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' !" ..
							" nvoverlaysink -e"}

It worked and displayed what was expected. Thanks a lot @AastaLLL and @jkjung13.