Using YOLO on Jetson TX2 and Econ System Cameras

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi, everyone. I recently got a Jetson TX2 and have successfully installed and tested the Econ System 6 camera system. I am able to view all 6 cameras with no issues. So now I am trying to run YOLO on only one or two of the cameras but not quite sure how to get started with it. I already have YOLO installed on the TX2 and can run it on saved images and videos successfully but I cannot get it to run on the camera streams. I am assuming the issue is because there is no way for it to identify which camera to look at. It also gives me this error of needing OpenCV to run the program, which I already have installed (I have OpenCV 3.3.0). Any advice would be super helpful.

2

You may check this https://jkjung-avt.github.io/yolov2/ for building yolo.

Be aware that yolo makefile relies on pkg-config for getting opencv include and lib paths.
Check that :

pkg-config --cflags opencv
pkg-config --libs opencv

give expected paths.

If ok or once you’ll have rebuilt yolo detector, if it complains about missing opencv lib when starting, you would try to add path to your opencv-3.3.0 libs into environment variable LD_LIBRARY_PATH. For example:

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/opencv-3.3.0/lib

See also https://github.com/AlexanderRobles21/YOLO-darknet-on-Jetson-TX2 for using gstreamer with nvcamerasrc (like for onboard camera) instead of V4L. Note this requires your opencv lib to have been built with gstreamer support.

Looking at the code, it displays on a 1352x1013 frame, so I suppose it expects a 4:3 frame aspect ratio, and with onboard camera I use

./darknet detector demo cfg/coco.data cfg/yolo.cfg yolo.weights "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)2592, height=(int)1944, format=(string)I420, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"

I see very different detection results depending on frame aspect ratio (but always same 2-3 fps with this model/weights, even when boosting my TX2).
You may adjust to your cameras capabilities.

3

@Honey_Patouceul: Hi, I tried doing all these steps but it is still giving me the same error but I think it’s because I am missing CUDA on my TX2. It was installed when I installed Jetpack successfully. But when I installed the drivers for the Econ Cameras and had to re-flash the TX2 (based on the instructions from Econ Systems), somehow the package got removed. Do you think this could be the reason? Just for reference, this is the instruction I followed to install opencv on TX2: How to Install OpenCV (3.4.0) on Jetson TX2

4

I have no experience with Econ on TX2, so it’s hard to help.
Probably Econ’s support can tell you if their install/flash process does/can install cuda and how to install within their setup.

You may also post the command that fails and the errors it logs, it may help someone to figure it out.

Furthermore, about missing cuda, what gives:

ls /var/cuda-repo-8-0-local
ls -al /usr/local/cuda
ls -al /usr/local/cuda/lib64/
5

@victoria.dgray
We have received your query from tech support. Our Business Development Manager to get in touch with you.

6

Hey Honey_Patouceul,

i use your code for running darknet on the onboard cam but i get following error after the weights are loaded:

mask_scale: Using default '1.000000'
Loading weights from yolo.weights...Done!
video file: nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)2592, height=(int)1944, format=(string)I420, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink
Couldn't connect to webcam.
: No such file or directory
darknet: ./src/utils.c:253: error: Assertion `0' failed.
Aborted (core dumped)

i have a jetson tx2 with JetPack3.2 flashed. So i have the OpenCv that comes with the new Jetpack.
Any suggestions why it doesn’t work?

7

Hi gustavvz,

JetPack-3.2 preview provides an opencv-3.3.1 version built without gstreamer suport which is mandatory for using onboard camera from opencv: [url]JetPack 3.2 — L4T R28.2 Developer Preview for Jetson TX2 - Jetson TX2 - NVIDIA Developer Forums.

I suggest you build your own opencv3 lib following this: [url]https://jkjung-avt.github.io/opencv3-on-tx2/[/url]

8

To start: I rebuild opencv 3.3.1 with CUDA nad GSTREAM support and now everything works perfectly fine. Also my tf object detection apis using various networks.

To Honey_Patoucel:

If you want to speed up to the 16fps that are possible with tiny-yolo and an usb-webcam just use this command:

./darknet detector demo cfg/voc.data cfg/tiny-yolo-voc.cfg tiny-yolo-voc.weights "nvcamerasrc ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"

if you just leave out the size formatting you dont lose fps.

Btw would be nice if you could eyplain step by step what happens between each ! some of those commands are not clear to me

9

Your pipeline string above is for using as argument from opencv. It is equivalent to this from shell with gst-launch:

nvcamerasrc ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! 'video/x-raw, format=(string)BGRx' ! videoconvert ! 'video/x-raw, format=(string)BGR' ! appsink

This enlights two types of elements:

  • the ones without quotes are gstreamer plugins. You can get the description of a plugin, its options and capabilities with
gst-inspect-1.0 plugin_to_inspect
  • the quoted ones are the caps (note that quotes are omitted when used from C/C++ code, as received when using yolo detector). It mainly defines a type/format between two plugins. If you don’t provide these, gstreamer will try to find itself if one (but maybe not the best) can match the output (source) capabilities of preceding plugin and input (sink) capabilities of following plugin.

So your pipeline first uses nvcamerasrc to get raw video from onboard camera. Note the “(memory:NVMM)”, as nvcamerasrc outputs to the memory space for GPU or ISP (in I420 or NV12 format).
Then nvvidconv is used to convert these into BGRx format in CPU memory space.
Then videoconvert is used to convert into BGR format in CPU memory space, as expected by opencv. Before opencv3.3, only BGR and gray8 were expected by opencv.
appsink is the gstreamer output for an application (here opencv-linked YOLO detector).

As written in post #2, I think that you will get worse detection results if you are not using a 4:3 frame ratio, so I would advise to keep 2592x1944 in the caps after nvcamerasrc.