opencv and see3cam-cu20 on jetson tk1

Hi,

I try to get a see3cam_cu20 camera from e-con Systems working on my jetson tk1 board. My configuration is as follow :

  • jetson tk1 board
  • Ubuntu linux 14.04 with linux for tegra 21.6
  • usb3 enabled
  • opencv 3.4.1

I installed the qtcam program from e-con Systems and it works fine if I run it with this command :

sudo QSG_RENDER_LOOP=basic qtcam

I also tried to use qv4l2 program and it works too.

Now, if I try to run this simple c++ program, the program is running but I only get a green screen. In addition, there is an error message “select timeout” (see attached picture).

#include <QCoreApplication>
#include <opencv2/core/core.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/opencv.hpp>
#include <opencv2/highgui.hpp>

#include <sstream>
#include <iostream>
#include <fstream>

using namespace cv;
using namespace std;

const int imageWidth = 640;
const int imageHeight = 480;

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    Mat frame(imageHeight,imageWidth, CV_32FC3);

    VideoCapture stream1(-1);
    if (!stream1.isOpened()) {
        cout << "cannot open camera";
        return a.exec();
    }
    stream1.set(CV_CAP_PROP_FRAME_WIDTH,imageWidth);
    stream1.set(CV_CAP_PROP_FRAME_HEIGHT,imageHeight);

    int framesPerSecond = stream1.get(CV_CAP_PROP_FPS);
    cout << "FPS = " << framesPerSecond << endl;
    cout << "Convert RGB : " << stream1.get(CV_CAP_PROP_CONVERT_RGB) << endl;
    cout << "Video size : " << stream1.get(CV_CAP_PROP_FRAME_WIDTH) << " x " << stream1.get(CV_CAP_PROP_FRAME_HEIGHT) << endl;

    cout << "OpenCV version : " << CV_VERSION << endl;

    namedWindow("seecam", CV_WINDOW_AUTOSIZE);
    bool stop = true;
    while(stop)
    {
        stream1 >> frame;
           if( frame.empty() ) {
               break;
               cout << "Frame is empty !" << endl;
           }

      imshow("seecam", frame);

      char character = waitKey(1000 / framesPerSecond);

      switch (character)
      {
          case 27:
              cout << "bye!" <<endl;
              stop = false;
              break;
      }
    }

    waitKey(0);

    return a.exec();
}

I can’t figure out where is the problem. Any idea ?

Thanks

Comment : I compile opencv from source with this option for video I/O :

Video I/O:
--     FFMPEG:                      YES
--       avcodec:                   YES (ver 54.35.1)
--       avformat:                  YES (ver 54.20.4)
--       avutil:                    YES (ver 52.3.0)
--       swscale:                   YES (ver 2.1.1)
--       avresample:                NO
--     libv4l/libv4l2:              NO
--     v4l/v4l2:                    linux/videodev2.h
--     gPhoto2:                     NO

Hi frederic.gauthier,

Are you able to open common usb webcam through your openCV3.4?

BTW, why is using “-1” in VideoCapture? I think the default one is 0.

Yes, I have a usb 2.0 webcam and I have no problem with it.
I though that -1 was used for the default camera and 0 for the device video0. The -1 works well with the webcam and I test 0 for the see3cam_cu20 (us3 3.1 gen 1) but no difference (when the driver of the camera is video0 of course).

Today, I recompiled opencv 3.4.1 with the option “-DWITH_LIBV4L=ON”. Now, it works but not all the time ! I can run my program 2 or 3 times in a row and the fourth time I will have a green screen. In this case, I have to unplug the camera and plug it in again to make it work again. The operation seems to me random because it can work 10 times in a row without problems and sometimes work only once.

I don’t know if it works now thanks to the recompilation because I do not remember if I disconnected and reconnected the camera yesterday during my tests.

Is there any log file or “trace” I can monitor to find the problem ?

frederic.gauthier,

This sounds opencv now is able to open the camera. If you hit this issue randomly, I wonder if qv4l2 program also has such error. Could you verify?

Also, do you see any error log in “dmesg” when green screen shows?

I also have errors with qv4l2 but I don’t need to unplug and reconnect the camera for it to work again. I also have error messages in the console. The first one is displayed each time I start a capture, and the second only appears from time to time :

First:

Error White Balance Temperature: Input/output error
v4l-convert: error short uyvy data frame

Second:

Error Brightness (2): Input/output error
Error Contrast (3): Input/output error
Error Saturation (6): Input/output error
Error Gain (5): Input/output error
Error White Balance Temperature: Input/output error
v4l-convert: error short uyvy data frame

For qv4l2 with dmesg I got this :

[ 4121.939615] gk20a gk20a.0: unrecognized ioctl cmd: 0x8008486d
[ 4121.963493] gk20a gk20a.0: unrecognized ioctl cmd: 0x8008486d
[ 4146.289133] tegra-xhci tegra-xhci: Firmware timestamp: 2014-09-16 02:10:07 UTC, Falcon state 0x20
[ 4146.784229] uvcvideo: Failed to query (SET_CUR) UVC control 2 on unit 2: -110 (exp. 2).
[ 4146.798447] uvcvideo: Failed to query (SET_CUR) UVC control 3 on unit 2: -71 (exp. 2).
[ 4146.816399] uvcvideo: Failed to query (SET_CUR) UVC control 7 on unit 2: -71 (exp. 2).
[ 4146.831714] uvcvideo: Failed to query (SET_CUR) UVC control 4 on unit 2: -71 (exp. 2).
[ 4146.840204] uvcvideo: Failed to query (GET_CUR) UVC control 10 on unit 2: -71 (exp. 2).
[ 4158.794421] uvcvideo: Non-zero status (-71) in video completion handler.

Now with my c++ code and opencv I actually have messages when it does not work (with dmesg):

[ 3218.168122] tegra-xhci tegra-xhci: Firmware timestamp: 2014-09-16 02:10:07 UTC, Falcon state 0x20
[ 3218.639460] uvcvideo: Failed to query (GET_CUR) UVC control 10 on unit 2: -110 (exp. 2).
[ 3218.991314] uvcvideo: Non-zero status (-71) in video completion handler.

I think that the problem comes from the line

tegra-xhci tegra-xhci: Firmware timestamp: 2014-09-16 02:10:07 UTC, Falcon state 0x20

but I don’t know what it means !

hello frederic.gauthier,

let’s narrow down the issue by using standard commands to access the camera sensor.
please check if you see same kernel failure message with below command, thanks

gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-raw,framerate=30/1,width=640,height=480 ! xvimagesink

Hello,

Here is the result of the command :

gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw,framerate=30/1,width=640,height=480 ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2865): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming task paused, reason not-negotiated (-4)
Execution ended after 0:00:00.009392130
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

and messages from dmesg :

[13683.576476] tegra-xhci tegra-xhci: Firmware timestamp: 2014-09-16 02:10:07 UTC, Falcon state 0x20
[13683.850987] usb 2-1: new SuperSpeed USB device number 3 using tegra-xhci
[13683.864406] usb 2-1: Parent hub missing LPM exit latency info.  Power management will be impacted.
[13683.866619] usb 2-1: New USB device found, idVendor=2560, idProduct=c120
[13683.866631] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[13683.866639] usb 2-1: Product: See3CAM_CU20
[13683.866645] usb 2-1: Manufacturer: e-con Systems
[13683.866652] usb 2-1: SerialNumber: 59050000
[13683.869828] uvcvideo: Found UVC 1.00 device See3CAM_CU20 (2560:c120)
[13683.871429] input: See3CAM_CU20 as /devices/platform/tegra-xhci/usb2/2-1/2-1:1.0/input/input5
[13684.697092] hid-generic 0003:2560:C120.0005: device has no listeners, quitting
[13695.814105] tegra-xhci tegra-xhci: Firmware timestamp: 2014-09-16 02:10:07 UTC, Falcon state 0x20
[13696.285794] uvcvideo: Failed to query (GET_DEF) UVC control 2 on unit 2: -110 (exp. 2).

Thanks,

Hello frederic,

Please ensure that the camera is indeed enumerated in USB super speed mode by using the following command :

ubuntu@tegra-ubuntu:~$ lsusb -d 2560:c120 -v | grep bcdUSB

The output should say 3.00

Please use the following command to stream the camera in gstreamer :

ubuntu@tegra-ubuntu:~$ gst-launch-1.0 v4l2src ! video/x-raw, format=UYVY, width=1280, height=720, framerate=30/1 ! xvimagesink

To find the other modes that are supported by the camera, please refer to the following link : https://www.e-consystems.com/wide-temperature-range-hdr-usb-cameraboard.asp or use the below command :

ubuntu@tegra-ubuntu:~$ v4l2-ctl --list-formats-ext

Use this list to configure the resolution and framerate correctly in the gstreamer pipeline/command.

Regarding OpenCV, we havent tested with the latest version of OpenCV yet. But your sample code looks a little bit complex involving 3 channel 32bit Floating point conversions. This could cause failures at different stages. To simplify the debugging process, could you please try the example code shared in the following link : https://www.e-consystems.com/Articles/Camera/opencv-jetson-using-13MP-MIPI-camera.asp

You will need to change the resolution used in the example as per the camera you test. Please share your results with the example code.

Hello,

Sorry for the late response but I was working on another project.

Here is the result of the command:

lsusb -d 2560:c120 -v | grep bcdUSB
Couldn't open device, some information will be missing
bcdUSB               3.00

When I connect the camera I have the following messages with " dmesg ":

[ 1894.692645] usb 2-1: New USB device found, idVendor=2560, idProduct=c120
[ 1894.692653] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1894.692659] usb 2-1: Product: See3CAM_CU20
[ 1894.692664] usb 2-1: Manufacturer: e-con Systems
[ 1894.692669] usb 2-1: SerialNumber: 59050000
[ 1894.694187] uvcvideo: Found UVC 1.00 device See3CAM_CU20 (2560:c120)
[ 1894.695689] input: See3CAM_CU20 as /devices/platform/tegra-xhci/usb2/2-1/2-1:1.0/input/input6

Apparently, I have no problem with gstreamer, the video always opens with this command

gst-launch-1.0 v4l2src ! video/x-raw, format=UYVY, width=1280, height=720, framerate=30/1 ! xvimagesink

.

I tested with the example code of your website:

#include <QCoreApplication>
#include <opencv2/core/core.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/opencv.hpp>
#include <opencv2/highgui.hpp>

#include <sstream>
#include <iostream>
#include <fstream>

using namespace cv;
using namespace std;

const int imageWidth = 640;
const int imageHeight = 480;

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    Mat image;

    VideoCapture cap;
    char keypressed;
    cap.open(-1);
    if( !cap.isOpened() )
    {
        cout << "***Could not initialize capturing...***\n";
        return -1;
    }
    cap.set(CV_CAP_PROP_FRAME_WIDTH,imageWidth);
    cap.set(CV_CAP_PROP_FRAME_HEIGHT,imageHeight);
    cout << "Width x Height = " << cap.get(CV_CAP_PROP_FRAME_WIDTH) << " x "
         << cap.get(CV_CAP_PROP_FRAME_HEIGHT) << "\n";
    namedWindow("Camera", CV_WINDOW_AUTOSIZE);
    for(;;)
    {
        cap >> image;
        if( image.empty() )
            break;
        imshow("Camera", image);
        keypressed = (char)waitKey(10);
        if( keypressed == 27 )
            break;
    }
    cout << "Type of image : " << image.type() << "\n";
    cap.release();
    return a.exec();
}

I have the same problem with this code. I can run it 2 or 3 times and then I get a green screen. I have to unplug the camera and plug it in again to make it work again.

Thanks,

hello frederic.gauthier,

if your camera sensor works with gstreamer, then there should be issues with the sample code.
could you monitor the system usage by tegrastats, and also sharing the kernel failure messages to us.
thanks

sudo ./tegrastats

Hello,

dmesg from the beginning in the attachment file.

dmesg and tegrastats when it fails :

[  253.700572] tegra-xhci tegra-xhci: Firmware timestamp: 2014-09-16 02:10:07 UTC, Falcon state 0x20
[  254.172525] uvcvideo: Failed to query (GET_CUR) UVC control 10 on unit 2: -110 (exp. 2).
[  257.584935] uvcvideo: Non-zero status (-71) in video completion handler.
RAM 615/1892MB (lfb 204x4MB) cpu [0%,off,off,off]@2065 EMC 3%@924 AVP 0%@204 VDE 120 GR3D 58%@72 EDP limit 0
RAM 615/1892MB (lfb 204x4MB) cpu [7%,off,off,off]@204 EMC 15%@204 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0
RAM 615/1892MB (lfb 204x4MB) cpu [11%,off,off,off]@204 EMC 15%@204 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0
RAM 615/1892MB (lfb 204x4MB) cpu [12%,off,off,off]@204 EMC 15%@204 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0
RAM 615/1892MB (lfb 204x4MB) cpu [12%,off,off,off]@204 EMC 15%@204 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0
RAM 615/1892MB (lfb 204x4MB) cpu [13%,off,off,off]@204 EMC 15%@204 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0
RAM 615/1892MB (lfb 204x4MB) cpu [18%,off,off,off]@564 EMC 18%@204 AVP 0%@204 VDE 120 GR3D 68%@72 EDP limit 0
RAM 616/1892MB (lfb 204x4MB) cpu [24%,off,off,off]@204 EMC 19%@204 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0
RAM 616/1892MB (lfb 204x4MB) cpu [10%,off,off,off]@204 EMC 16%@204 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0
RAM 616/1892MB (lfb 204x4MB) cpu [11%,off,off,off]@204 EMC 14%@204 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0
RAM 616/1892MB (lfb 204x4MB) cpu [11%,off,off,off]@204 EMC 14%@204 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0
RAM 616/1892MB (lfb 204x4MB) cpu [8%,off,off,off]@1224 EMC 4%@600 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0
RAM 616/1892MB (lfb 204x4MB) cpu [7%,off,off,off]@204 EMC 14%@204 AVP 0%@204 VDE 120 GR3D 0%@72 EDP limit 0

Then dmesg when I unplug and plug the camera again :

[  453.179506] tegra-xhci tegra-xhci: Firmware timestamp: 2014-09-16 02:10:07 UTC, Falcon state 0x20
[  453.375389] usb 2-1: USB disconnect, device number 2
[  455.621690] usb 2-1: new SuperSpeed USB device number 3 using tegra-xhci
[  455.633738] usb 2-1: Parent hub missing LPM exit latency info.  Power management will be impacted.
[  455.635163] usb 2-1: New USB device found, idVendor=2560, idProduct=c120
[  455.635173] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  455.635180] usb 2-1: Product: See3CAM_CU20
[  455.635187] usb 2-1: Manufacturer: e-con Systems
[  455.635193] usb 2-1: SerialNumber: 59050000
[  455.636854] uvcvideo: Found UVC 1.00 device See3CAM_CU20 (2560:c120)
[  455.638476] input: See3CAM_CU20 as /devices/platform/tegra-xhci/usb2/2-1/2-1:1.0/input/input5
[  456.617164] hid-generic 0003:2560:C120.0005: device has no listeners, quitting

dmesg and tegrastats when it works :

[  579.829702] systemd-hostnamed[2499]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname!
[  681.776403] tegra-xhci tegra-xhci: Firmware timestamp: 2014-09-16 02:10:07 UTC, Falcon state 0x20
[  682.249288] uvcvideo: Failed to query (GET_DEF) UVC control 2 on unit 2: -110 (exp. 2).
RAM 656/1892MB (lfb 194x4MB) cpu [0%,0%,off,off]@2065 EMC 25%@924 AVP 0%@204 VDE 120 GR3D 11%@396 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [73%,off,off,off]@2065 EMC 24%@924 AVP 0%@204 VDE 120 GR3D 38%@180 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [76%,off,off,off]@2065 EMC 24%@924 AVP 0%@204 VDE 120 GR3D 28%@180 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [78%,off,off,off]@2065 EMC 24%@924 AVP 0%@204 VDE 120 GR3D 51%@180 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [78%,off,off,off]@2065 EMC 24%@924 AVP 0%@204 VDE 120 GR3D 51%@180 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [76%,off,off,off]@2065 EMC 24%@924 AVP 0%@204 VDE 120 GR3D 64%@108 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [78%,off,off,off]@2065 EMC 24%@924 AVP 0%@204 VDE 120 GR3D 54%@180 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [76%,off,off,off]@2065 EMC 24%@924 AVP 0%@204 VDE 120 GR3D 66%@108 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [80%,off,off,off]@2065 EMC 24%@924 AVP 0%@204 VDE 120 GR3D 36%@180 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [83%,off,off,off]@2065 EMC 24%@924 AVP 0%@204 VDE 120 GR3D 52%@180 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [81%,off,off,off]@2065 EMC 25%@924 AVP 0%@204 VDE 120 GR3D 49%@180 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [78%,off,off,off]@2065 EMC 24%@924 AVP 0%@204 VDE 120 GR3D 37%@180 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [77%,off,off,off]@2065 EMC 25%@924 AVP 0%@204 VDE 120 GR3D 35%@180 EDP limit 0
RAM 656/1892MB (lfb 194x4MB) cpu [79%,off,off,off]@2065 EMC 28%@924 AVP 0%@204 VDE 120 GR3D 86%@180 EDP limit 0

Thanks,
dmesg_log.txt (56.6 KB)

Out of date, but looking at opencv source, it seems -1 is for /dev/video, so it is not correct on Jetson unless you’ve made a /dev/video link to /dev/video0.

Hi,

While researching another subject, I found the answer to my question.

I just disabled USB auto-suspend (see https://elinux.org/Jetson/Performance) and now I’m able to run my program again and again!

echo -1 > /sys/module/usbcore/parameters/autosuspend

thanks
Fred

Nice! Thanks for sharing!