jetson-inference with OpenCV camera input?

I installed OpenCV 3.2 so I can use the Jetson TX2 builtin camera.

I also installed jetson-inference so I can run detectnet-camera.

Can I modify it to use cv2::VideoCapture from OpenCV 3.2 instead of gstCamera to get camera input?

How can I do this?

Hi wiany11,

built-in camera outputs incompatible format for Opencv3.2, which only supports BGR and gray8, so the answer to your question may be “No”. Please use gst pipeline if you want to use opencv and built-in cam

I can read from the biltin camera.

OpenCV4Tegra (from OpenCV2) could not utilize the camera but OpenCV 3.2 can.

Would you please at least specify the datatype of image that jetson-inference can take?

Hi,

Why do you want to open camera with OpenCV? Do you need some preprocess via OpenCV?

Jetson_inference pipeline should like this:
DMA buffer -> register with EGL -> CUDA memory

For OpenCV,
cv::mat use CPU memory, and cv::gpu:GpuMat use GPU memory

So if you want to feed data directly into TensorRT, you need to use cv::gpu::GpuMat and make sure your function support GpuMat.

One trick is to simplify this is to allocate a unified memory. (shared memory between GPU and CPU)
Feed this into tensorRT and also use this memory to read the camera input.

@AastaLLL

Hello.

I do not know about gstCamera…

I am try to figure out jetson-inference’s samples.

If I change flip-method to odd number, e.g., 3, I got broken images.
(flip-method was 2.)
(BTW, I rotate my monitor to the left using xrandr.)

I also do not know how to remove margins… haha…

So I want to go with a little bit more familiar OpenCV.

So as long as I convert cv::Mat to cv::cuda::GpuMat, can I use jetson-inference’s detectNet?

YES.

TensorRT needs GPU memory. If you store an image in GPU memory, you can run it with TensorRT.

@AastaLLL

Thanks! I will try!

@AastaLLL

I tried to pass cv::cuda::GpuMat to net->Detect like:

cv::Mat frame;
cv::cuda::GpuMat frame_gpu;

camera.read(frame);
frame_gpu.upload(frame);
float* imgCUDA = frame_gpu.ptr<float>();

int imgWidth = frame_gpu.size().width;
int imgHeight = frame_gpu.size().height;
int numBoundingBoxes = maxBoxes;
if (net->Detect(imgCUDA, imgWidth, imgHeight,
                bbCPU, &numBoundingBoxes, confCPU)) {
    std::cout << numBoundingBoxes << std::endl;
} else {
    std::cout << "failed to classify" << std::endl;
}

But I got the followings:

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 …

[GIE] layer deploy_transform input reformatter 0 - 0.000000 ms
[GIE] layer deploy_transform - 0.000000 ms
[GIE] layer conv1/7x7_s2 + conv1/relu_7x7 - 0.000000 ms
[GIE] layer pool1/3x3_s2 - 0.000000 ms
[GIE] layer pool1/norm1 - 0.000000 ms
[GIE] layer conv2/3x3_reduce + conv2/relu_3x3_reduce - 0.000000 ms
[GIE] layer conv2/3x3 + conv2/relu_3x3 - 0.000000 ms
[GIE] layer conv2/norm2 - 0.000000 ms
[GIE] layer pool2/3x3_s2 - 0.000000 ms
[GIE] layer inception_3a/1x1 + inception_3a/relu_1x1 || inception_3a/3x3_reduce + inception_3a/relu_3x3_reduce || inception_3a/5x5_reduce + inception_3a/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_3a/3x3 + inception_3a/relu_3x3 - 0.000000 ms
[GIE] layer inception_3a/5x5 + inception_3a/relu_5x5 - 0.000000 ms
[GIE] layer inception_3a/pool - 0.000000 ms
[GIE] layer inception_3a/pool_proj + inception_3a/relu_pool_proj - 0.000000 ms
[GIE] layer inception_3a/1x1 copy - 0.000000 ms
[GIE] layer inception_3b/1x1 + inception_3b/relu_1x1 || inception_3b/3x3_reduce + inception_3b/relu_3x3_reduce || inception_3b/5x5_reduce + inception_3b/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_3b/3x3 + inception_3b/relu_3x3 - 0.000000 ms
[GIE] layer inception_3b/5x5 + inception_3b/relu_5x5 - 0.000000 ms
[GIE] layer inception_3b/pool - 0.000000 ms
[GIE] layer inception_3b/pool_proj + inception_3b/relu_pool_proj - 0.000000 ms
[GIE] layer inception_3b/1x1 copy - 0.000000 ms
[GIE] layer pool3/3x3_s2 - 0.000000 ms
[GIE] layer inception_4a/1x1 + inception_4a/relu_1x1 || inception_4a/3x3_reduce + inception_4a/relu_3x3_reduce || inception_4a/5x5_reduce + inception_4a/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_4a/3x3 + inception_4a/relu_3x3 - 0.000000 ms
[GIE] layer inception_4a/5x5 + inception_4a/relu_5x5 - 0.000000 ms
[GIE] layer inception_4a/pool - 0.000000 ms
[GIE] layer inception_4a/pool_proj + inception_4a/relu_pool_proj - 0.000000 ms
[GIE] layer inception_4a/1x1 copy - 0.000000 ms
[GIE] layer inception_4b/1x1 + inception_4b/relu_1x1 || inception_4b/3x3_reduce + inception_4b/relu_3x3_reduce || inception_4b/5x5_reduce + inception_4b/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_4b/3x3 + inception_4b/relu_3x3 - 0.000000 ms
[GIE] layer inception_4b/5x5 + inception_4b/relu_5x5 - 0.000000 ms
[GIE] layer inception_4b/pool - 0.000000 ms
[GIE] layer inception_4b/pool_proj + inception_4b/relu_pool_proj - 0.000000 ms
[GIE] layer inception_4b/1x1 copy - 0.000000 ms
[GIE] layer inception_4c/1x1 + inception_4c/relu_1x1 || inception_4c/3x3_reduce + inception_4c/relu_3x3_reduce || inception_4c/5x5_reduce + inception_4c/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_4c/3x3 + inception_4c/relu_3x3 - 0.000000 ms
[GIE] layer inception_4c/5x5 + inception_4c/relu_5x5 - 0.000000 ms
[GIE] layer inception_4c/pool - 0.000000 ms
[GIE] layer inception_4c/pool_proj + inception_4c/relu_pool_proj - 0.000000 ms
[GIE] layer inception_4c/1x1 copy - 0.000000 ms
[GIE] layer inception_4d/1x1 + inception_4d/relu_1x1 || inception_4d/3x3_reduce + inception_4d/relu_3x3_reduce || inception_4d/5x5_reduce + inception_4d/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_4d/3x3 + inception_4d/relu_3x3 - 0.000000 ms
[GIE] layer inception_4d/5x5 + inception_4d/relu_5x5 - 0.000000 ms
[GIE] layer inception_4d/pool - 0.000000 ms
[GIE] layer inception_4d/pool_proj + inception_4d/relu_pool_proj - 0.000000 ms
[GIE] layer inception_4d/1x1 copy - 0.000000 ms
[GIE] layer inception_4e/1x1 + inception_4e/relu_1x1 || inception_4e/3x3_reduce + inception_4e/relu_3x3_reduce || inception_4e/5x5_reduce + inception_4e/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_4e/3x3 + inception_4e/relu_3x3 - 0.000000 ms
[GIE] layer inception_4e/5x5 + inception_4e/relu_5x5 - 0.000000 ms
[GIE] layer inception_4e/pool - 0.000000 ms
[GIE] layer inception_4e/pool_proj + inception_4e/relu_pool_proj - 0.000000 ms
[GIE] layer inception_4e/1x1 copy - 0.000000 ms
[GIE] layer inception_5a/1x1 + inception_5a/relu_1x1 || inception_5a/3x3_reduce + inception_5a/relu_3x3_reduce || inception_5a/5x5_reduce + inception_5a/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_5a/3x3 + inception_5a/relu_3x3 - 0.000000 ms
[GIE] layer inception_5a/5x5 + inception_5a/relu_5x5 - 0.000000 ms
[GIE] layer inception_5a/pool - 0.000000 ms
[GIE] layer inception_5a/pool_proj + inception_5a/relu_pool_proj - 0.000000 ms
[GIE] layer inception_5a/1x1 copy - 0.000000 ms
[GIE] layer inception_5b/1x1 + inception_5b/relu_1x1 || inception_5b/3x3_reduce + inception_5b/relu_3x3_reduce || inception_5b/5x5_reduce + inception_5b/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_5b/3x3 + inception_5b/relu_3x3 - 0.000000 ms
[GIE] layer inception_5b/5x5 + inception_5b/relu_5x5 - 0.000000 ms
[GIE] layer inception_5b/pool - 0.000000 ms
[GIE] layer inception_5b/pool_proj + inception_5b/relu_pool_proj - 0.000000 ms
[GIE] layer inception_5b/1x1 copy - 0.000000 ms
[GIE] layer cvg/classifier - 0.000000 ms
[GIE] layer cvg/classifier output reformatter 0 - 0.000000 ms
[GIE] layer bbox/regressor - 0.000000 ms
[GIE] layer bbox/regressor output reformatter 0 - 0.000000 ms
[GIE] cudnnEngine.cpp (387) - Cuda Error in execute: 4
[GIE] detectNet::Classify() – failed to execute tensorRT context
failed to classify

What should I do to pass OpenCV camera input (cv::cuda::GpuMat) to net->Detect of jetson-inference?

Hi,

  1. Pointer of GpuMat is at frame_gpu.data
  2. Passing the pointer into TensorRT directly.
    https://github.com/dusty-nv/jetson-inference/blob/master/detectNet.cpp#L247

Hi,
I’m trying to run a simple opencv cpp code for edge detection on Jetson TX2
And getting the error:

/usr/include/opencv2/stitching.hpp:234:13: error: ‘Status’ does not name a type
     CV_WRAP Status estimateTransform(InputArrayOfArrays images);
             ^
/usr/include/opencv2/stitching.hpp:244:5: error: ‘Status’ does not name a type
     Status estimateTransform(InputArrayOfArrays images, const std::vector<std::vector<Rect> > &rois);
     ^
/usr/include/opencv2/stitching.hpp:247:13: error: ‘Status’ does not name a type
     CV_WRAP Status composePanorama(OutputArray pano);
             ^
/usr/include/opencv2/stitching.hpp:259:5: error: ‘Status’ does not name a type
     Status composePanorama(InputArrayOfArrays images, OutputArray pano);
     ^
/usr/include/opencv2/stitching.hpp:262:13: error: ‘Status’ does not name a type
     CV_WRAP Status stitch(InputArrayOfArrays images, OutputArray pano);
             ^
/usr/include/opencv2/stitching.hpp:270:5: error: ‘Status’ does not name a type
     Status stitch(InputArrayOfArrays images, const std::vector<std::vector<Rect> > &rois, OutputArray pano);
     ^
/usr/include/opencv2/stitching.hpp:279:5: error: ‘Status’ does not name a type
     Status matchImages();
     ^
/usr/include/opencv2/stitching.hpp:280:5: error: ‘Status’ does not name a type
     Status estimateCameraParams();
     ^

The code is:

#include "opencv2/opencv.hpp"
using namespace cv;

int main(int, char**)
{
	VideoCapture cap(-1); // open the default camera
	if(!cap.isOpened())  // check if we succeeded
		return -1;
	cap.set(CV_CAP_PROP_FRAME_WIDTH,1920);
	cap.set(CV_CAP_PROP_FRAME_HEIGHT,1080);
	Mat edges;
	namedWindow("edges",CV_WINDOW_AUTOSIZE);
	for(;;)
	{
		cap >> frame; // get a new frame from camera
		if (frame.empty())
			break;
		cvtColor(frame, edges, CV_BGR2GRAY);
		GaussianBlur(edges, edges, Size(7,7), 1.5, 1.5);
		Canny(edges, edges, 0, 30, 3);
		imshow("edges", edges);
		if((char)waitKey(10) == 27) break;
	}
	// the camera will be deinitialized automatically in VideoCapture destructor
	return 0;
}

Can I know the complete process of installation of OpenCV 3.2 on Jetson TX2 and an example of OpenCv code?
Also need to know how to run the code.

Hi,

You can check this page for OpenCV installation and testing:
http://dev.t7.ai/jetson/opencv/

Thanks.

I am also having the same issue. I have tried to pass the pointer to the data in the GPU mat but I get the same error.

Code:

Mat inputImageCPU;

inputImageCPU = imread("Opencv_Net/Sidewalk.png", CV_LOAD_IMAGE_COLOR);
cuda::GpuMat inputImageGPU(inputImageCPU);

float* inCUDA = (float*)inputImageGPU.data;

int imgWidth = inputImageGPU.size().width;
int imgHeight = inputImageGPU.size().height;

// allocate segmentation overlay output buffer
float* outCPU  = NULL;
float* outCUDA = NULL;

if( !cudaAllocMapped((void**)&outCPU, (void**)&outCUDA, imgWidth * imgHeight * sizeof(float) * 4) )
{
  printf("segnet-camera:  failed to allocate CUDA memory for output image\n");
  return 0;
}

// process image overlay
if( !net->Overlay((float*)inCUDA, (float*)outCUDA, imgWidth, imgHeight) )
{
  printf("segnet-console:  failed to process segmentation overlay.\n");
} else {
    printf("SUCCESS!!!!\n");
}

output:

[GIE]  TensorRT version 3.0, build 3000
[GIE]  attempting to open cache file Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel.2.tensorcache
[GIE]  loading network profile from cache... Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel.2.tensorcache
[GIE]  platform has FP16 support.
[GIE]  Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel loaded
[GIE]  CUDA engine context initialized with 2 bindings
[GIE]  Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel input  binding index:  0
[GIE]  Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel input  dims (b=2 c=3 h=360 w=640) size=5529600
[cuda]  cudaAllocMapped 5529600 bytes, CPU 0x101540000 GPU 0x101540000
[GIE]  Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel output 0 score_fr  binding index:  1
[GIE]  Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel output 0 score_fr  dims (b=2 c=21 h=6 w=14) size=14112
[cuda]  cudaAllocMapped 14112 bytes, CPU 0x101a90000 GPU 0x101a90000
Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel initialized.
[cuda]  cudaAllocMapped 336 bytes, CPU 0x101340200 GPU 0x101340200
[GIE]  segNet outputs -- s_w 14  s_h 6  s_c 21
[cuda]  cudaAllocMapped 84 bytes, CPU 0x101340400 GPU 0x101340400
segNet -- class 00  color 0 0 0 255
segNet -- class 01  color 0 141 0 255
segNet -- class 02  color 176 176 0 255
segNet -- class 03  color 0 0 206 255
segNet -- class 04  color 0 255 255 255
segNet -- class 05  color 146 0 0 255
segNet -- loaded 6 class colors
'egNet -- class 00  label 'Background
'egNet -- class 01  label 'Building
'egNet -- class 02  label 'Other
'egNet -- class 03  label 'Vegitation
'egNet -- class 04  label 'Vehicle
'egNet -- class 05  label 'Drivable
segNet -- loaded 6 class labels
[cuda]  cudaAllocMapped 3686400 bytes, CPU 0x101c90000 GPU 0x101c90000
[GIE]  cudnnEngine.cpp (419) - Cuda Error in execute: 4
[GIE]  segNet::Overlay() -- failed to execute tensorRT context
segnet-console:  failed to process segmentation overlay.

Hi,

Could you check if your error comes from TensorRT inference or CUDA overlay?
Thanks.

@winay11 Reference to the answer–#63 of “https://devtalk.nvidia.com/default/topic/1007313/jetson-tx2/how-to-build-the-objection-detection-framework-ssd-with-tensorrt-on-tx2-/5”.
“Most common issue is memory leakage.
Could you check if there is any non-free memory in the implementation?” by nvidia official answer AastaLLL
i also got the same error"cudnnEngine.cpp (387) - Cuda Error in execute: 4"
At last I solved the problem by:
add the memory release code after inference through execute() and copy gpu data to cpu through cudaMemcpy().
my code is:
CHECK(!cudaFree(buffers[0]));
CHECK(!cudaFree(buffers[1]));

Hope it would be useful for you and other guys!