display delay and very slow display frame rate

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
#1

Hi，I have few questions about image display on Xavier.

Two problems: display delay and very slow display frame rate.

We use OpenCV function “VideoCapture::read” for image capture and OpenCV function “imshow” for image display. Between the two functions we use Caffe to process image. The time of processing image we print is 70ms, in other words, the processing frame rate is 1000/70=13fps, but display frame rate is 1fps and delay is about 3s.

This is beyond our understanding, could you help us solve these problems?

Thanks!

#2

402561047,

For performance issue, it would be better if you could share your app or a simplified one that can reproduce the issue.

#3

Thanks for your reply！

Our code as below:

#include <time.h>
#include <stdlib.h>
#include <stdio.h>
#include

#include “ObjectDetect.h”
#include “opencv2/core/core.hpp”
#include “opencv2/highgui/highgui.hpp”
#include “opencv2/imgproc/imgproc.hpp”

int main(int argc, char **argv)
{
//initialize object detection
char conPath[100] = “/home/nvidia/tools/models/obj.txt”;
HC::InitObjectDec(conPath);

//open camera
cv::VideoCapture cap(0);
if (!cap.isOpened()) {
	std::cout << "Failed to open video: " << std::endl;
}
//set camera para
cap.set(CV_CAP_PROP_FRAME_WIDTH, 1920);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 1080);

//image from camera
cv::Mat img;


while (true)
{
    //capture image
	cap>>img;

    //image for show
	cv::Mat showIm = img.clone();
    
    //calculate time
	clock_t startT, endT;
	startT = clock();

	unsigned char* data_obj = (unsigned char*)img.data;
	HC::object_info_obj tmpObj;
    tmpObj.objectNum = 0;
    //object detection
	HC::ObjectDectect(data_obj, img.cols, img.rows, 2,(HC::object_info_obj*)(&tmpObj));
	
	endT = (double)(1000 * (clock() - startT) / CLOCKS_PER_SEC);
	std::cout << "Processing time : "<<endT<<"ms" << std::endl;
	
    //draw image
	char scoreStr[10] = "";
	for (int i = 0; i<tmpObj.objectNum; i++)
	{
		std::string label;
		cv::Rect tmpRt;
		tmpRt.x = tmpObj.objectRts[i].x;
		tmpRt.y = tmpObj.objectRts[i].y;
		tmpRt.width = tmpObj.objectRts[i].width;
		tmpRt.height = tmpObj.objectRts[i].height;
        
        sprintf(scoreStr, "%f", tmpObj.confidence[i]);

		cv::rectangle(showIm, tmpRt, cv::Scalar(0, 255, 205), 2, 4, 0);
		cv::putText(showIm, scoreStr, cv::Point(tmpObj.objectRts[i].x,tmpObj.objectRts[i].y), 2, 0.5, cv::Scalar(255, 255));
	}
    //show image
	imshow("result-1", showIm);
	cv::waitKey(5);
}

HC::ObjectDectectFree();
system("pause");
getchar();
return 0;

}

#4

I see there are some library needed, could you share the way to set up environment?

It would be better if you could just share a tarball.

#5

OpenCV and DarkNet library are needed. But, the weights file size is about 240MB, so which way receive these files is convenient for you？

#6

You may want to check the FPS of your camera. People have reported that the frame rate must be explicitly set, it defaults to ~ 3FPS. E.g.

cap.set(cv2.CAP_PROP_FPS,30)
#7

Thanks for your reply. “cap.set(CV_CAP_PROP_FPS,30)” is added to the app, but it do not work.

#8

402561047,

If this is the issue of display frame, I think it may be related to input format.

You could try to remove the process line by line and see which function is causing the rendering become slow.

#9

Only one line for image process: HC::ObjectDectect(data_obj, img.cols, img.rows, 2,(HC::object_info_obj*)(&tmpObj));

the line : std::cout << "Processing time : "<<endT<<“ms” << std::endl;

Processing time : 70ms

As shown above, Processing time is 70ms, but the display frame rate is very low, and delay is about 3s.

#10

402561047,

Will you see low frame rate if you feed a dummy input data which is not being processed?

#11

It may work, but failing to read back the new value.

Not sure about your problem, but if your processing function takes 70 ms, with additional putText, imshow (you may create a window once before), and your 5 ms delay in waitKey, let’s say about 80ms, then you won’t be able to process all frames if you use a higher framerate than 12 fps. So you may have actually set fps to 30, but missing 2 frames on 3 and not seeing improvement. You may try first a low framerate :

cap.set(cv2.CAP_PROP_FPS, 3)

and increase later if it works.

If not yet done, check all cores are up with nvpmodel and boost clocks.
When it runs, you may check for bottleneck with tegrastats.

Could you comment the caffe part and check what is max framerate you can acheive, first without adding rectangles and text, then adding a fixed number of rectangles and text (same as typical objects you label with detector), or add timings on various parts of the loop (including waitKey) ?