aron.h
August 1, 2023, 4:03pm
1
• Hardware Platform (Jetson / GPU) • DeepStream Version • JetPack Version (valid for Jetson only) • TensorRT Version • Issue Type( questions, new requirements, bugs)
We get cv::Mat frames using this OpenCV gstreamer pipeline:filesrc location="./video.mp4" ! qtdemux ! h264parse ! queue ! nvv4l2decoder ! queue ! nvvideoconvert ! video/x-raw,format=BGRx ! videorate max-rate=30 ! videoscale ! video/x-raw,format=BGRx,width=1920,height=1080 ! queue ! videoconvert ! video/x-raw,format=BGR ! appsink
We specify a model using Yolov7::Yolov7 after which we pass the cv::Mat frames to Yolov7::preProcess and then run Yolov7::infer and Yolov7::PostProcess.
The inference works and everything runs fine at this point (around 31 seconds to process a 31 second video).
After further examination, the culprit lies somewhere within the method enqueueV2 mentioned in Yolov7.cppNvInfer.h and NvInferRuntime.h to NvInferImpl.h.class VExecutionContext : public VRoot has the methodvirtual bool enqueueV2(void* const* bindings, cudaStream_t stream, cudaEvent_t* inputConsumed) noexcept = 0;
Any idea of why this is happening?
We are using the library called Yolov7 made by an Nvidia employee.
yuweiw
August 2, 2023, 7:48am
7
Could you attach the whole pipeline including the nvinfer part? You can also get the latency and memory loading by referring: memory latency .
aron.h
August 2, 2023, 8:11am
8
Here is a boiled down and hard coded vesion of what is relevant.
#include <opencv2/opencv.hpp>
#include "Yolov7.h"
int main()
{
Yolov7 yolov7("model.engine");
int frame_count = 0;
std::vector<std::vector<std::vector<float>>> nmsresults;
cv::VideoCapture cap = cv::VideoCapture("filesrc location=video.mp4 ! qtdemux ! h264parse ! queue ! nvv4l2decoder ! queue ! nvvideoconvert ! video/x-raw,format=BGRx ! videorate max-rate=30 ! videoscale ! video/x-raw,format=BGRx,width=1920,height=1080 ! queue ! videoconvert ! video/x-raw,format=BGR ! appsink", cv::CAP_GSTREAMER);
while (frame_count < 1000)
{
cv::Mat cap_mat;
std::vector<cv::Mat> cap_mat_vec;
cap >> cap_mat;
if (!cap_mat.data) { break; }
cap_mat_vec.push_back(cap_mat);
yolov7.preProcess(cap_mat_vec);
yolov7.infer();
nmsresults = yolov7.PostProcess();
cap_mat_vec.clear();
frame_count++;
}
cap.release();
return 0;
}
For the sake of expediting results, the following changes can be made to the Yolov7 library.
CMakeLists.txt lines 73 to 81:add_executable and target_link_librariesdetect, video_detect and validate_coco
Yolov7.h Line 96:static int Yolov::DrawBoxesonGraph(... to static int DrawBoxesonGraph(...
aron.h
August 2, 2023, 9:14am
9
The exampe code above had some bugs, but they have been edited out now.
yuweiw
August 2, 2023, 10:57am
10
Could you get the information in your environment? Maybe the loading is too heavy when spin up another thread that does the same thing in parallel.
aron.h
August 2, 2023, 6:15pm
11
Most of the measurements mentioned in that post assume that one is using deepstream sample apps.
One of the posts suggested to download and run nvmemstat.py while running our application to get more information. After minor tweaking of the python script (a case of ‘.decode’ expecting a string, but getting an int),
Next, I ran our own application with 1 input source.
PID: 10072 18:12:06 Total used hardware memory: 0 hardware memory: 0.0000 KiB VmSize: 32326.2305 MiB VmRSS: 417.5469 MiB RssFile: 210.5898 MiB RssAnon: 215.2500 MiB lsof: 333
Finally, I ran our own application with 6 input sources.
PID: 7742 18:07:09 Total used hardware memory: 0 hardware memory: 0.0000 KiB VmSize: 32418.0156 MiB VmRSS: 574.6719 MiB RssFile: 273.9023 MiB RssAnon: 311.7773 MiB lsof: 333
I am not sure what I should be looking for in these reports?
yuweiw
August 3, 2023, 3:03am
12
It’s werid that Total used hardware memory: 0 hardware memory: 0.0000 KiB in your environment. As you said before, I traced it’s origin via NvInfer.h and NvInferRuntime.h to NvInferImpl.h.. So the possible reason is that gpu loading is heavy. Below is the printing in my environment：
PID: 59003 10:56:11 Total used hardware memory: 1261736K hardware memory: 825.2676 MiB VmSize: 35084.5234 MiB VmRSS: 2940.2344 MiB RssFile: 1324.2695 MiB RssAnon: 1609.5273 MiB lsof: 743
Also could you describe how the code you posted works step by step and the specific time comparison data?
aron.h
August 3, 2023, 8:48am
13
I tried running the python script from outside our docker and with sudo priviliges.
I ran our own application with 1 input source.
PID: 2145718 09:29:18 Total used hardware memory: 195808K hardware memory: 193.1445 MiB VmSize: 32375.0156 MiB VmRSS: 495.5430 MiB RssFile: 236.0508 MiB RssAnon: 260.1133 MiB lsof: 332
Then I ran our own application with 6 input sources.
PID: 2168169 09:37:59 Total used hardware memory: 204000K hardware memory: 199.1445 MiB VmSize: 32482.0078 MiB VmRSS: 612.9297 MiB RssFile: 273.8477 MiB RssAnon: 341.0820 MiB lsof: 333
yuweiw
August 3, 2023, 9:31am
14
Just from the log, the 6 sources need 2341196K memory and the 1 source needs 944156K. What is the memory of your machine?
Its an AGX Orin 32GB. Here’s a screenshot from JTOP
yuweiw
August 3, 2023, 11:00am
16
OK. Could you provide the detailed steps to reproduce the issue, including the code, compiling and run?
aron.h
August 3, 2023, 3:21pm
17
Our codebase is way to large to post here.
1 Like
yuweiw
August 4, 2023, 8:34am
18
OK. Thanks for providing the demo. We will analyze it as soon as possible.
@yuweiw any luck replicating the issue ?
yuweiw
August 7, 2023, 1:37am
20
Yes. I can reproduce the problem in my environment. It may take some time to analyze this issue.
aron.h:
I traced it’s origin via NvInfer.h and NvInferRuntime.h to NvInferImpl.h.class VExecutionContext : public VRoot has the methodvirtual bool enqueueV2(void* const* bindings, cudaStream_t stream, cudaEvent_t* inputConsumed) noexcept = 0;
How did you come to this conclusion above? Based on my results, the time for all three processes(preprocess, infer, postprocess) has increased.
aron.h
August 7, 2023, 2:34pm
21
yuweiw:
Yes. I can reproduce the problem in my environment. It may take some time to analyze this issue.
aron.h:
I traced it’s origin via NvInfer.h and NvInferRuntime.h to NvInferImpl.h.class VExecutionContext : public VRoot has the methodvirtual bool enqueueV2(void* const* bindings, cudaStream_t stream, cudaEvent_t* inputConsumed) noexcept = 0;
How did you come to this conclusion above? Based on my results, the time for all three processes(preprocess, infer, postprocess) has increased.
After I assertained that the project took too long to run, I started commenting out the content of it, starting form the bottom and working my way up.infer() call, the time it took to run the project immediately dropped down to acceptable levels (ie. slightly more than the duration of the video file).infer() call and started commenting out lines inside the Yolov7::infer method itself.if (!mContext->enqueueV2(mBindingArray.data(), mStream.get(), NULL)) was causing the slow-down.
yuweiw
August 8, 2023, 3:10am
22
You can use the jtop or tegrastats tool to observing the usage rate of GPU. It is full when run 6 threads. And the preprocess and postprocess use OpenCV which use the gpu to process data. This will also slow down the process speed.
What’s the way around this ? Surely orin agx can run inference on 6 1080p videos without slow down. The slow down starts from just 2 1080p videos.
yuweiw
August 8, 2023, 8:57am
24
How did this conclusion come from? The demo you used uses the CPU to process data. If you want to improve efficiency, you can use the DeepStream, like:
source1
source2
source3 -------------> streammux ---------------> inference ------->.......
....
sourcen
aron.h
August 9, 2023, 9:10am
25
We did try using gstreamer plugins in a similar fashion as your diagram above shows.this post, we had some problems with nvinfer so at the end of that dialogue, we were advised to use the libary Yolov7 to do the inference.
