jetson-inference with OpenCV camera input?

wiany11 · August 17, 2017, 3:05am

I installed OpenCV 3.2 so I can use the Jetson TX2 builtin camera.

I also installed jetson-inference so I can run detectnet-camera.

Can I modify it to use cv2::VideoCapture from OpenCV 3.2 instead of gstCamera to get camera input?

How can I do this?

WayneWWW · August 17, 2017, 8:16am

Hi wiany11,

built-in camera outputs incompatible format for Opencv3.2, which only supports BGR and gray8, so the answer to your question may be “No”. Please use gst pipeline if you want to use opencv and built-in cam

wiany11 · August 17, 2017, 9:09am

I can read from the biltin camera.

OpenCV4Tegra (from OpenCV2) could not utilize the camera but OpenCV 3.2 can.

Would you please at least specify the datatype of image that jetson-inference can take?

AastaLLL · August 17, 2017, 10:13am

Hi,

Why do you want to open camera with OpenCV? Do you need some preprocess via OpenCV?

Jetson_inference pipeline should like this:
DMA buffer → register with EGL → CUDA memory

For OpenCV,
cv::mat use CPU memory, and cv::gpu:GpuMat use GPU memory

So if you want to feed data directly into TensorRT, you need to use cv::gpu::GpuMat and make sure your function support GpuMat.

One trick is to simplify this is to allocate a unified memory. (shared memory between GPU and CPU)
Feed this into tensorRT and also use this memory to read the camera input.

wiany11 · August 18, 2017, 2:25am

@AastaLLL

Hello.

I do not know about gstCamera…

I am try to figure out jetson-inference’s samples.

If I change flip-method to odd number, e.g., 3, I got broken images.
(flip-method was 2.)
(BTW, I rotate my monitor to the left using xrandr.)

I also do not know how to remove margins… haha…

So I want to go with a little bit more familiar OpenCV.

So as long as I convert cv::Mat to cv::cuda::GpuMat, can I use jetson-inference’s detectNet?

AastaLLL · August 18, 2017, 2:49am

YES.

TensorRT needs GPU memory. If you store an image in GPU memory, you can run it with TensorRT.

wiany11 · August 18, 2017, 2:54am

@AastaLLL

Thanks! I will try!

wiany11 · August 18, 2017, 8:15am

@AastaLLL

I tried to pass cv::cuda::GpuMat to net->Detect like:

cv::Mat frame;
cv::cuda::GpuMat frame_gpu;

camera.read(frame);
frame_gpu.upload(frame);
float* imgCUDA = frame_gpu.ptr<float>();

int imgWidth = frame_gpu.size().width;
int imgHeight = frame_gpu.size().height;
int numBoundingBoxes = maxBoxes;
if (net->Detect(imgCUDA, imgWidth, imgHeight,
                bbCPU, &numBoundingBoxes, confCPU)) {
    std::cout << numBoundingBoxes << std::endl;
} else {
    std::cout << "failed to classify" << std::endl;
}

But I got the followings:

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 …

[GIE] layer deploy_transform input reformatter 0 - 0.000000 ms
[GIE] layer deploy_transform - 0.000000 ms
[GIE] layer conv1/7x7_s2 + conv1/relu_7x7 - 0.000000 ms
[GIE] layer pool1/3x3_s2 - 0.000000 ms
[GIE] layer pool1/norm1 - 0.000000 ms
[GIE] layer conv2/3x3_reduce + conv2/relu_3x3_reduce - 0.000000 ms
[GIE] layer conv2/3x3 + conv2/relu_3x3 - 0.000000 ms
[GIE] layer conv2/norm2 - 0.000000 ms
[GIE] layer pool2/3x3_s2 - 0.000000 ms
[GIE] layer inception_3a/1x1 + inception_3a/relu_1x1 || inception_3a/3x3_reduce + inception_3a/relu_3x3_reduce || inception_3a/5x5_reduce + inception_3a/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_3a/3x3 + inception_3a/relu_3x3 - 0.000000 ms
[GIE] layer inception_3a/5x5 + inception_3a/relu_5x5 - 0.000000 ms
[GIE] layer inception_3a/pool - 0.000000 ms
[GIE] layer inception_3a/pool_proj + inception_3a/relu_pool_proj - 0.000000 ms
[GIE] layer inception_3a/1x1 copy - 0.000000 ms
[GIE] layer inception_3b/1x1 + inception_3b/relu_1x1 || inception_3b/3x3_reduce + inception_3b/relu_3x3_reduce || inception_3b/5x5_reduce + inception_3b/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_3b/3x3 + inception_3b/relu_3x3 - 0.000000 ms
[GIE] layer inception_3b/5x5 + inception_3b/relu_5x5 - 0.000000 ms
[GIE] layer inception_3b/pool - 0.000000 ms
[GIE] layer inception_3b/pool_proj + inception_3b/relu_pool_proj - 0.000000 ms
[GIE] layer inception_3b/1x1 copy - 0.000000 ms
[GIE] layer pool3/3x3_s2 - 0.000000 ms
[GIE] layer inception_4a/1x1 + inception_4a/relu_1x1 || inception_4a/3x3_reduce + inception_4a/relu_3x3_reduce || inception_4a/5x5_reduce + inception_4a/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_4a/3x3 + inception_4a/relu_3x3 - 0.000000 ms
[GIE] layer inception_4a/5x5 + inception_4a/relu_5x5 - 0.000000 ms
[GIE] layer inception_4a/pool - 0.000000 ms
[GIE] layer inception_4a/pool_proj + inception_4a/relu_pool_proj - 0.000000 ms
[GIE] layer inception_4a/1x1 copy - 0.000000 ms
[GIE] layer inception_4b/1x1 + inception_4b/relu_1x1 || inception_4b/3x3_reduce + inception_4b/relu_3x3_reduce || inception_4b/5x5_reduce + inception_4b/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_4b/3x3 + inception_4b/relu_3x3 - 0.000000 ms
[GIE] layer inception_4b/5x5 + inception_4b/relu_5x5 - 0.000000 ms
[GIE] layer inception_4b/pool - 0.000000 ms
[GIE] layer inception_4b/pool_proj + inception_4b/relu_pool_proj - 0.000000 ms
[GIE] layer inception_4b/1x1 copy - 0.000000 ms
[GIE] layer inception_4c/1x1 + inception_4c/relu_1x1 || inception_4c/3x3_reduce + inception_4c/relu_3x3_reduce || inception_4c/5x5_reduce + inception_4c/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_4c/3x3 + inception_4c/relu_3x3 - 0.000000 ms
[GIE] layer inception_4c/5x5 + inception_4c/relu_5x5 - 0.000000 ms
[GIE] layer inception_4c/pool - 0.000000 ms
[GIE] layer inception_4c/pool_proj + inception_4c/relu_pool_proj - 0.000000 ms
[GIE] layer inception_4c/1x1 copy - 0.000000 ms
[GIE] layer inception_4d/1x1 + inception_4d/relu_1x1 || inception_4d/3x3_reduce + inception_4d/relu_3x3_reduce || inception_4d/5x5_reduce + inception_4d/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_4d/3x3 + inception_4d/relu_3x3 - 0.000000 ms
[GIE] layer inception_4d/5x5 + inception_4d/relu_5x5 - 0.000000 ms
[GIE] layer inception_4d/pool - 0.000000 ms
[GIE] layer inception_4d/pool_proj + inception_4d/relu_pool_proj - 0.000000 ms
[GIE] layer inception_4d/1x1 copy - 0.000000 ms
[GIE] layer inception_4e/1x1 + inception_4e/relu_1x1 || inception_4e/3x3_reduce + inception_4e/relu_3x3_reduce || inception_4e/5x5_reduce + inception_4e/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_4e/3x3 + inception_4e/relu_3x3 - 0.000000 ms
[GIE] layer inception_4e/5x5 + inception_4e/relu_5x5 - 0.000000 ms
[GIE] layer inception_4e/pool - 0.000000 ms
[GIE] layer inception_4e/pool_proj + inception_4e/relu_pool_proj - 0.000000 ms
[GIE] layer inception_4e/1x1 copy - 0.000000 ms
[GIE] layer inception_5a/1x1 + inception_5a/relu_1x1 || inception_5a/3x3_reduce + inception_5a/relu_3x3_reduce || inception_5a/5x5_reduce + inception_5a/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_5a/3x3 + inception_5a/relu_3x3 - 0.000000 ms
[GIE] layer inception_5a/5x5 + inception_5a/relu_5x5 - 0.000000 ms
[GIE] layer inception_5a/pool - 0.000000 ms
[GIE] layer inception_5a/pool_proj + inception_5a/relu_pool_proj - 0.000000 ms
[GIE] layer inception_5a/1x1 copy - 0.000000 ms
[GIE] layer inception_5b/1x1 + inception_5b/relu_1x1 || inception_5b/3x3_reduce + inception_5b/relu_3x3_reduce || inception_5b/5x5_reduce + inception_5b/relu_5x5_reduce - 0.000000 ms
[GIE] layer inception_5b/3x3 + inception_5b/relu_3x3 - 0.000000 ms
[GIE] layer inception_5b/5x5 + inception_5b/relu_5x5 - 0.000000 ms
[GIE] layer inception_5b/pool - 0.000000 ms
[GIE] layer inception_5b/pool_proj + inception_5b/relu_pool_proj - 0.000000 ms
[GIE] layer inception_5b/1x1 copy - 0.000000 ms
[GIE] layer cvg/classifier - 0.000000 ms
[GIE] layer cvg/classifier output reformatter 0 - 0.000000 ms
[GIE] layer bbox/regressor - 0.000000 ms
[GIE] layer bbox/regressor output reformatter 0 - 0.000000 ms
[GIE] cudnnEngine.cpp (387) - Cuda Error in execute: 4
[GIE] detectNet::Classify() – failed to execute tensorRT context
failed to classify

What should I do to pass OpenCV camera input (cv::cuda::GpuMat) to net->Detect of jetson-inference?

AastaLLL · August 21, 2017, 3:25am

Hi,

Pointer of GpuMat is at frame_gpu.data
Passing the pointer into TensorRT directly.
[url]https://github.com/dusty-nv/jetson-inference/blob/master/detectNet.cpp#L247[/url]

pharatekomal · October 30, 2017, 11:33am

Hi,
I’m trying to run a simple opencv cpp code for edge detection on Jetson TX2
And getting the error:

/usr/include/opencv2/stitching.hpp:234:13: error: ‘Status’ does not name a type
     CV_WRAP Status estimateTransform(InputArrayOfArrays images);
             ^
/usr/include/opencv2/stitching.hpp:244:5: error: ‘Status’ does not name a type
     Status estimateTransform(InputArrayOfArrays images, const std::vector<std::vector<Rect> > &rois);
     ^
/usr/include/opencv2/stitching.hpp:247:13: error: ‘Status’ does not name a type
     CV_WRAP Status composePanorama(OutputArray pano);
             ^
/usr/include/opencv2/stitching.hpp:259:5: error: ‘Status’ does not name a type
     Status composePanorama(InputArrayOfArrays images, OutputArray pano);
     ^
/usr/include/opencv2/stitching.hpp:262:13: error: ‘Status’ does not name a type
     CV_WRAP Status stitch(InputArrayOfArrays images, OutputArray pano);
             ^
/usr/include/opencv2/stitching.hpp:270:5: error: ‘Status’ does not name a type
     Status stitch(InputArrayOfArrays images, const std::vector<std::vector<Rect> > &rois, OutputArray pano);
     ^
/usr/include/opencv2/stitching.hpp:279:5: error: ‘Status’ does not name a type
     Status matchImages();
     ^
/usr/include/opencv2/stitching.hpp:280:5: error: ‘Status’ does not name a type
     Status estimateCameraParams();
     ^

The code is:

#include "opencv2/opencv.hpp"
using namespace cv;

int main(int, char**)
{
	VideoCapture cap(-1); // open the default camera
	if(!cap.isOpened())  // check if we succeeded
		return -1;
	cap.set(CV_CAP_PROP_FRAME_WIDTH,1920);
	cap.set(CV_CAP_PROP_FRAME_HEIGHT,1080);
	Mat edges;
	namedWindow("edges",CV_WINDOW_AUTOSIZE);
	for(;;)
	{
		cap >> frame; // get a new frame from camera
		if (frame.empty())
			break;
		cvtColor(frame, edges, CV_BGR2GRAY);
		GaussianBlur(edges, edges, Size(7,7), 1.5, 1.5);
		Canny(edges, edges, 0, 30, 3);
		imshow("edges", edges);
		if((char)waitKey(10) == 27) break;
	}
	// the camera will be deinitialized automatically in VideoCapture destructor
	return 0;
}

Can I know the complete process of installation of OpenCV 3.2 on Jetson TX2 and an example of OpenCv code?
Also need to know how to run the code.

AastaLLL · October 31, 2017, 2:16am

Hi,

You can check this page for OpenCV installation and testing:
http://dev.t7.ai/jetson/opencv/

Thanks.

ssullivan · February 26, 2018, 4:14pm

I am also having the same issue. I have tried to pass the pointer to the data in the GPU mat but I get the same error.

Code:

Mat inputImageCPU;

inputImageCPU = imread("Opencv_Net/Sidewalk.png", CV_LOAD_IMAGE_COLOR);
cuda::GpuMat inputImageGPU(inputImageCPU);

float* inCUDA = (float*)inputImageGPU.data;

int imgWidth = inputImageGPU.size().width;
int imgHeight = inputImageGPU.size().height;

// allocate segmentation overlay output buffer
float* outCPU  = NULL;
float* outCUDA = NULL;

if( !cudaAllocMapped((void**)&outCPU, (void**)&outCUDA, imgWidth * imgHeight * sizeof(float) * 4) )
{
  printf("segnet-camera:  failed to allocate CUDA memory for output image\n");
  return 0;
}

// process image overlay
if( !net->Overlay((float*)inCUDA, (float*)outCUDA, imgWidth, imgHeight) )
{
  printf("segnet-console:  failed to process segmentation overlay.\n");
} else {
    printf("SUCCESS!!!!\n");
}

output:

[GIE]  TensorRT version 3.0, build 3000
[GIE]  attempting to open cache file Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel.2.tensorcache
[GIE]  loading network profile from cache... Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel.2.tensorcache
[GIE]  platform has FP16 support.
[GIE]  Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel loaded
[GIE]  CUDA engine context initialized with 2 bindings
[GIE]  Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel input  binding index:  0
[GIE]  Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel input  dims (b=2 c=3 h=360 w=640) size=5529600
[cuda]  cudaAllocMapped 5529600 bytes, CPU 0x101540000 GPU 0x101540000
[GIE]  Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel output 0 score_fr  binding index:  1
[GIE]  Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel output 0 score_fr  dims (b=2 c=21 h=6 w=14) size=14112
[cuda]  cudaAllocMapped 14112 bytes, CPU 0x101a90000 GPU 0x101a90000
Opencv_Net/UnityNet/snapshot_iter_2024.caffemodel initialized.
[cuda]  cudaAllocMapped 336 bytes, CPU 0x101340200 GPU 0x101340200
[GIE]  segNet outputs -- s_w 14  s_h 6  s_c 21
[cuda]  cudaAllocMapped 84 bytes, CPU 0x101340400 GPU 0x101340400
segNet -- class 00  color 0 0 0 255
segNet -- class 01  color 0 141 0 255
segNet -- class 02  color 176 176 0 255
segNet -- class 03  color 0 0 206 255
segNet -- class 04  color 0 255 255 255
segNet -- class 05  color 146 0 0 255
segNet -- loaded 6 class colors
'egNet -- class 00  label 'Background
'egNet -- class 01  label 'Building
'egNet -- class 02  label 'Other
'egNet -- class 03  label 'Vegitation
'egNet -- class 04  label 'Vehicle
'egNet -- class 05  label 'Drivable
segNet -- loaded 6 class labels
[cuda]  cudaAllocMapped 3686400 bytes, CPU 0x101c90000 GPU 0x101c90000
[GIE]  cudnnEngine.cpp (419) - Cuda Error in execute: 4
[GIE]  segNet::Overlay() -- failed to execute tensorRT context
segnet-console:  failed to process segmentation overlay.

AastaLLL · March 2, 2018, 7:43am

Hi,

Could you check if your error comes from TensorRT inference or CUDA overlay?
Thanks.

chunxianwang · June 25, 2018, 11:53am

@winay11 Reference to the answer–#63 of “https://devtalk.nvidia.com/default/topic/1007313/jetson-tx2/how-to-build-the-objection-detection-framework-ssd-with-tensorrt-on-tx2-/5”.
“Most common issue is memory leakage.
Could you check if there is any non-free memory in the implementation?” by nvidia official answer AastaLLL
i also got the same error"cudnnEngine.cpp (387) - Cuda Error in execute: 4"
At last I solved the problem by:
add the memory release code after inference through execute() and copy gpu data to cpu through cudaMemcpy().
my code is:
CHECK(!cudaFree(buffers[0]));
CHECK(!cudaFree(buffers[1]));
Hope it would be useful for you and other guys!

Topic		Replies	Views
Opencv Face Detection Poor Performance with jetson nano Jetson Nano opencv	51	14164	October 14, 2021
JETPACK 4.2 - JETSON TX2 - OpenCV 3.4.6 - GStreamer - Problem with on-board camera Jetson TX2	11	2031	October 18, 2021
X264 and TensorRT sudden reboot (MJPG encoder not affected, but not fast enough) on Jetson Orin Nano Jetson Orin Nano tensorrt , jetson-inference , gstreamer , jetson	52	793	June 18, 2024
openCv + detectNet in python Jetson Nano camera , python	11	2336	October 15, 2021
Get wrong result when I using tensorRT to do inference, am I wrong to use ? Jetson TX2	18	2982	October 18, 2021
OpenCV error Jetson Nano camera , opencv , jetson-inference , gstreamer , python	18	9279	October 15, 2021
How does one encode and stream detectnet object detection video over network using the dusty-jetson-inference repo? Jetson TX2	15	3852	October 18, 2021
On board camera Jetson TX2 gstreamer	14	2158	October 18, 2021
OpenCV : the function detectMultiScale hangs / crashes on Xavier Jetson AGX Xavier	14	3070	October 18, 2021
video output does not show up because, [OpenGL] failed to create X11 Window imagenet-camera: failed to create openGL display Jetson TX2	10	6699	October 18, 2021

jetson-inference with OpenCV camera input?

Related topics