Multi-threading using Python on TX2

Hi guys,

I’m trying to use multi-threading to increase the speed of my program and more precisely the fps for a real-time application.

To make it simple here is how I try to make things work:

  • I capture an image from the live-feed of the oboard camera (thread 1)
  • I apply some treatment on this image (thread 2)
  • I show the image continuously (thread 3)

Here are the 2 scripts (Thread definition and Main):

from threading import Thread, RLock
import cv2
from AphidFunction import *

lock=RLock()

class VideoShow(Thread):

    def __init__(self,grabbed,frame):
        Thread.__init__(self)
        self.frame=frame
        self.grabbed=grabbed

    def run(self):
        while self.grabbed == True:
            with lock: 
                cv2.imshow('Frame',self.frame)

class VideoGet(Thread):

    def __init__(self,source):
        Thread.__init__(self)
        self.stream=cv2.VideoCapture(source)
        (self.grabbed,self.frame)=self.stream.read()

def run(self):
        with lock:
            (self.grabbed, self.frame)=self.stream.read() 

class detection(Thread):

    def __init__(self,frame):
        Thread.__init__(self)
        self.frame=frame

def run(self):
        while True:
            with lock: 
                hsv=image_treatment(self.frame)
                img=find_aphids(hsv)
                self.frame=img

And the Main:

#!/usr/bin/env python
from VideoThread import *

thread_get=VideoGet(source)
thread_get.start()

frame1=thread_get.frame
grabbed1=thread_get.grabbed

thread_detection=detection(frame1)
thread_detection.start()

thread_show=VideoShow(grabbed1,frame1)
thread_show.start()

while True:

    frame = thread_get.frame
    ok=thread_get.grabbed
    thread_detection.frame=frame
    thread_show.grabbed=ok
    thread_show.frame=thread_detection.frame

Unfortunately I have this recurring error I don’t get…I don’t even use the locateROI function:

error: /home/nvidia/OpenCV3.4/opencv-3.4.0/modules/core/src/matrix.cpp:991: error: (-215) dims <= 2 && step[0] > 0 in function locateROI

Any idea ?

I also tried to put the while loop of the Main directly in the Thread script but the same error occurs.

PS: AphidsFunction is a home-made script describing image_treatment and find_aphids function. They work well.

Hi,

A common issue is that the required image is invalid.
Could you check the functionality of image source first?

def __init__(self,source):
        Thread.__init__(self)
        self.stream=cv2.VideoCapture(source)
        (self.grabbed,self.frame)=self.stream.read()

Thanks.

Hi,
here is what I tried to check functionality:

thread_get=VideoGet(source)
thread_get.start()

while True:
    frame1=thread_get.frame
    cv2.imshow('Frame',frame1)

with this source (the onboard camera I always used and that works fine in other script):

source="nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)640, height=(int)360,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"

And here is the error I get:

VIDEOIO ERROR: V4L: device nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)640, height=(int)360,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels

Hi,

Do you build OpenCV from source?

Please noticed that the default OpenCV package doesn’t enable V4L support.
You need to build it from source.

Here is an automatically script for your reference:
https://github.com/AastaNV/JEP/blob/master/script/install_opencv3.4.0.sh

Thanks.

Yes I did using this procedure: https://jkjung-avt.github.io/opencv3-on-tx2/

From what I can see the difference is coming from these dependencies that my procedure didn’t indicate:

  • v4l-utils
  • qv4l2
  • and v4l2ucp

But I notice that in my procedure I have enable WITH_LIBV4L=ON in the cmake command, so should I just install the lacking packages or do I have to re-build it from source using your script ?

Thanks

  1. Make sure your “non-threaded” code works first. That way you are sure your hardware (camera) and all dependent libraries are good.

  2. As AastaLLL already mentioned, (I guess) your detection thread throws out an error probably because you call it with ‘None’ as the input image. That is, your detection thread starts processing images before the VideoGet thread captures the 1st image from the camera.