TX2 with FLIR Thermal Camera throwing an error (on Orbitty Board)

I am using a TX2 on an Orbitty carrier board with a FLIR Boson 320-240 thermal camera configured as a webcam.

When I run a real-time object detection app with a regular webcam is works fine, but with the thermal camera it show the following :

libv4l2: error set_fmt gave us a different result then try_fmt!
VIDEOIO ERROR: libv4l unable convert to requested pixfmt
VIDEOIO ERROR: V4L: Initial Capture Error: Unable to load initial memory buffers.
Segmentation fault (core dumped)

it works with “cheese” as a webcam so I know the TX2 can see the camera and display it.

I figure the problem has to do with the thermal camera sending a different type of color profile and the current program I am using isn’t converting it, but I have no idea how to check any of this, so I am looking for some input as far as next steps.

You have to check the APP to use the pixel format that this sensor can support.

I am using the following app: https://github.com/naisy/realtime_object_detection

The interesting part is, it works on my Xavier but doesn’t work on the TX2. I think this is because of the specific open CV install and maybe the g-streamer compatibility?

According to FLIR this is what they say about the USB webcam:

Boson is capable of providing digital data as a USB Video Class (UVC) compliant device. Two output options are provided. Note the options are not selected via the CCI but rather by the video capture or viewing software selected by the user. The options are:
■ Pre-AGC (16-bit): The output is linearly proportional to the flux incident on each pixel in the array; output resolution is 320x256 for the 320 configuration, 640x512 for the 640 configuration. Note that AGC settings, zoom settings, and color-encoding settings have no effect on the output signal at this tap point. This option is identified with a UVC video format 4CC code of “Y16 ” (16-bit uncompressed greyscale image)
■ Post-Colorize, YCbCrb: The output is transformed to YCbCr color space using the specified color palette (see Section 6.7). Resolution is 640x512 for both the 320 and 640 configurations. Three options are provided, identified via the UVC video format 4CC code:
• I420: 8 bit Y plane followed by 8 bit 2x2 subsampled U and V planes
• NV12: 8-bit Y plane followed by an interleaved U/V plane with 2x2 subsampling
• NV21: same as NV12 except reverse order of U and V planes

I am guessing I need to make sure that one of the three 4CC codes are compatible, I just have no idea where to look?

Looking at what is below I am guessing open CV is trying to open the 16bit monochrome. Googling the error it seems like most people were working with monochrome images, so I guess I am looking for a way to force it to use the color version?

https://www.flir.com/support-center/oem/is-there-a-way-to-maximize-the-video-display-on-the-boson-app-for-windows-pc-to-full-screen/

When I would use a simple open CV program to display the webcam it would give me the same error, but then display the video, my guess is it tries the monochrome gets and error then moves onto the color version. I am looking for a way for open CV to not fail when it gets the wrong format.

When I run the code below prior to installing pyyaml it just doesn’t work and says:

OpenCV Error: Unspecified error (GStreamer: unable to start pipeline
) in cvCaptureFromCAM_GStreamer, file /home/nvidia/src/opencv-3.4.0/modules/videoio/src/cap_gstreamer.cpp, line 890
VIDEOIO(cvCreateCapture_GStreamer(CV_CAP_GSTREAMER_V4L2, reinterpret_cast<char *>(index))): raised OpenCV exception:

/home/nvidia/src/opencv-3.4.0/modules/videoio/src/cap_gstreamer.cpp:890: error: (-2) GStreamer: unable to start pipeline
 in function cvCaptureFromCAM_GStreamer

OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /home/nvidia/src/opencv-3.4.0/modules/imgproc/src/color.cpp, line 11111
Traceback (most recent call last):
  File "test.py", line 11, in <module>
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
cv2.error: /home/nvidia/src/opencv-3.4.0/modules/imgproc/src/color.cpp:11111: error: (-215) scn == 3 || scn == 4 in function cvtColor

But after it gives the same errors as above but then displays an image.

Open CV webcam display code

import numpy as np
import cv2

cap = cv2.VideoCapture(0)

while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()

    # Our operations on the frame come here
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Display the resulting frame
    cv2.imshow('frame',gray)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

Using that same code I can not get the FLIR to display on the Xavier, but it does work with the above program.

We were able to figure it out thanks to this: https://github.com/FLIR/BosonUSB/issues/13

We replaced the open CV video capture with this :

pipeline = "v4l2src device=/dev/video0 ! video/x-raw,format=BGR ! videoscale ! video/x-raw,width=640,height=480 ! videoconvert ! appsink"
        self.vid = cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER)