# Performance optimization help

**URL:** <https://forums.developer.nvidia.com/t/performance-optimization-help/67567>\
**Category:** Jetson TX2\
**Created:** [November 19, 2018, 7:04am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567 "2018-11-19T07:04:10Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![dumbogeorge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dumbogeorge](https://forums.developer.nvidia.com/u/dumbogeorge)\
**Post date:** [November 19, 2018, 7:04am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/1 "2018-11-19T07:04:10Z")

</div>

Hi Folks,

We have following pipeline, where we are processing video from each camera on GPU/CPU and then selecting output from either camera to be encoded.

```auto
Cam1 -->--> GPU --> CPU (frame processing, and data annotation ) -->
                                                                   |
                                                                   |--> select either cam source --> encoder --> encoded bitstream
                                                                   | 
Cam2 -->--> GPU --> CPU (frame processing, and data annotation ) -->

```

When we operate each camera at 1080/30 - some of our CV (aaNewOCVConsumerThread, aaCamCaptureThread) routines eat up significant percentage of cycles. However when we operate each camera at 1080/60, then, some of nvidia resource manager API seem to take significant percentage of cycles (report from nvidia system profiler 3.9 attached).

Could someone please help - get to bottom of this ? Why nvidia resource manager takes long cycles ? What is going on in v4l2convert.so ? Is there a way to avoid some sort of conversion going on here ?

Thanks  
 ![](https://sea2.discourse-cdn.com/nvidia/home/nvidia/Downloads/PerfScreenShot1.png)

---

<div class="post-metadata">

**Author:** ![dumbogeorge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dumbogeorge](https://forums.developer.nvidia.com/u/dumbogeorge)\
**Post date:** [November 19, 2018, 7:04am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/2 "2018-11-19T07:04:35Z")

</div>

What is best way to attach an image here ?  
Thanks

---

<div class="post-metadata">

**Author:** ![DaneLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/danelll/32/14043_2.png) [@DaneLLL](https://forums.developer.nvidia.com/u/DaneLLL)\
**Post date:** [November 19, 2018, 8:38am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/3 "2018-11-19T08:38:23Z")

</div>

Hi,  
Please break down to see which stage uses high loading of video converter.

```auto
Cam1 -->--> GPU --> CPU (frame processing, and data annotation ) -->

Cam2 -->--> GPU --> CPU (frame processing, and data annotation ) -->

```

```auto
Cam1 -->--> GPU -->

Cam2 -->--> GPU -->

```

```auto
Cam1 -->-->

Cam2 -->-->

```

You should be able to upload an image by clicking the image icon, no?

---

<div class="post-metadata">

**Author:** ![dumbogeorge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dumbogeorge](https://forums.developer.nvidia.com/u/dumbogeorge)\
**Post date:** [November 19, 2018, 8:53am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/4 "2018-11-19T08:53:57Z")

</div>

I get just “ ![]()” when I click image icon.

Thanks

---

<div class="post-metadata">

**Author:** ![DaneLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/danelll/32/14043_2.png) [@DaneLLL](https://forums.developer.nvidia.com/u/DaneLLL)\
**Post date:** [November 19, 2018, 8:59am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/5 "2018-11-19T08:59:56Z")

</div>

> [@](#):
>
> I get just “ ![]()” when I click image icon.
> 
> Thanks

Do you use Chrome?

---

<div class="post-metadata">

**Author:** ![dumbogeorge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dumbogeorge](https://forums.developer.nvidia.com/u/dumbogeorge)\
**Post date:** [November 19, 2018, 9:45am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/6 "2018-11-19T09:45:08Z")

</div>

Yes. Any otherway to share image - will dropbox work ?

---

<div class="post-metadata">

**Author:** ![DaneLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/danelll/32/14043_2.png) [@DaneLLL](https://forums.developer.nvidia.com/u/DaneLLL)\
**Post date:** [November 20, 2018, 2:22am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/7 "2018-11-20T02:22:07Z")

</div>

Hi,  
You may use any free online services. We suggest you break down the pipeline to clarify which stage the high loading of video converter comes from.

---

<div class="post-metadata">

**Author:** ![dumbogeorge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dumbogeorge](https://forums.developer.nvidia.com/u/dumbogeorge)\
**Post date:** [November 21, 2018, 4:12am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/8 "2018-11-21T04:12:29Z")

</div>

Please find data pictures here -

> **[Dropbox](https://www.dropbox.com/sh/6ov6fuox4ialqjr/AABtM-ISYKa_v733pubJ2-KAa?dl=0)**

PerfScreenShot1.png refers to entire pipeline described in #1.

PerfScreenShot2.png refers to following pipeline

```auto
Cam1 (1080p @ 60) ----> renderer

```

Please help optimize driver calls in v4l2convert.so calls.

Thanks

---

<div class="post-metadata">

**Author:** ![DaneLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/danelll/32/14043_2.png) [@DaneLLL](https://forums.developer.nvidia.com/u/DaneLLL)\
**Post date:** [November 21, 2018, 5:22am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/9 "2018-11-21T05:22:16Z")

</div>

Hi,  
v4l2convert.so is a 3rdpary prebuilt library in v4l2 frameworks, not owned by NVIDIA. From the result, it looks to be triggered by high frame rate. In 60fps, you will see 60 capture request via v4l2\_ioctl.

One thing you can check is the sensor mode and output resolution. Take onboard camera as an example, it supports 3 sensor modes:

```auto
PRODUCER: [0] W=2592 H=1944
PRODUCER: [1] W=2592 H=1458
PRODUCER: [2] W=1280 H=720

```

If you run

```auto
09_camera_jpeg_capture$ ./camera_jpeg_capture --disable-jpg --sensor-mode 0 --pre-res=1280x720 --cap-time 15

```

It puts loading on HW video converter doing 2592x1944 → 1280x720 conversion.

Otherwise, if you run

```auto
09_camera_jpeg_capture$ ./camera_jpeg_capture --disable-jpg --sensor-mode 2 --pre-res=1280x720 --cap-time 15

```

Making sensor mode same as output resolution can save the loading on HW video converter.

---

<div class="post-metadata">

**Author:** ![dumbogeorge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dumbogeorge](https://forums.developer.nvidia.com/u/dumbogeorge)\
**Post date:** [November 21, 2018, 9:46am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/10 "2018-11-21T09:46:57Z")

</div>

Hi DaneLLL

We are actually using native sensor resolution as our final output resolution, in our code. How can we zero-in on what is triggering an unsolicited conversion, and how can we get rid of it ?

I do not understand why would high fps trigger it ? Any idea what calls could be going on in v4l2convert.so ? or is there a quick and easy way for owner of this library to offer a “no operation” version of those APIs ? That could help in determining if those calls are unnecessary or not ?

Furthermore in order to imitate the simplified command line - of

```auto
Cam1 ---> renderer

```

I tried argus\_camera app - which seems much more optimal. From the profiler result it seem that app is NOT making use of v4l2convert.so ? How can I find out what is leading upto call to v4l2convert.so in my code and eliminate it ?

Thanks,

---

<div class="post-metadata">

**Author:** ![DaneLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/danelll/32/14043_2.png) [@DaneLLL](https://forums.developer.nvidia.com/u/DaneLLL)\
**Post date:** [November 22, 2018, 1:31am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/11 "2018-11-22T01:31:38Z")

</div>

Hi,  
The implementation of Argus SW stacks is:

```auto
sensor -> ioctl(VIDIOC_DQBUF) -> captured frames in raw format -> VI/ISP -> frames in I420/NV12 format

```

That’s why it shows higher loading in v4l2convert.so in 60fps. When comparing to 30fps, capture requests are double.

v4l2convert.so is in open source v4l2 stacks [v4l-utils.git - media (V4L2, DVB and IR) applications and libraries](https://git.linuxtv.org/v4l-utils.git)

---

<div class="post-metadata">

**Author:** ![dumbogeorge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dumbogeorge](https://forums.developer.nvidia.com/u/dumbogeorge)\
**Post date:** [November 22, 2018, 4:44am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/12 "2018-11-22T04:44:49Z")

</div>

Hi DaneLLL

Would the pipeline that you have given, i.e.

```auto
sensor -> ioctl(VIDIOC_DQBUF) -> captured frames in raw format -> VI/ISP -> frames in I420/NV12 format

```

apply to your app - argus\_camera (from tegra\_multimedia\_api) ?

I uploaded profile data from this app (without any video capture, but running 1080 @ 60 fps) at

```auto
https://www.dropbox.com/s/3ot6t7azlrdd7o7/PerScreenShot-argus-camera-app.png?dl=0

```

and I do not see any v4l2convert call here. Would it be easy for me to copy this examples for my pipeline - i.e. what I described in #1 ?

---

<div class="post-metadata">

**Author:** ![DaneLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/danelll/32/14043_2.png) [@DaneLLL](https://forums.developer.nvidia.com/u/DaneLLL)\
**Post date:** [November 22, 2018, 6:30am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/13 "2018-11-22T06:30:53Z")

</div>

Hi,  
The pipeline is built in libargus.so. Your application should run the same capture pipeline.

We have source code of argus\_camera at tegra\_multimedia\_api\argus\apps\camera. You may compare it with your application.  
You may also download [v4l-utils.git - media (V4L2, DVB and IR) applications and libraries](https://git.linuxtv.org/v4l-utils.git) , put debug prints, and rebuild v4l2convert.so to check where the loading is from. The branch is stable-1.0

---

<div class="post-metadata">

**Author:** ![dumbogeorge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dumbogeorge](https://forums.developer.nvidia.com/u/dumbogeorge)\
**Post date:** [December 1, 2018, 10:35am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/14 "2018-12-01T10:35:32Z")

</div>

Hi DaneLLL

I studied / copied tegra\_multimedia\_api/argus/apps/camera. Compared it with I have implemented. I am pretty much using same argus APIs which are being used in argus\_camera app. While I am still working on this particular performance issue, I would like to ask a pipeline question related to mapping of buffers. I am suspecting that it could be leading to few unwanted conversion api calls.

My pipeline looks like -

```auto
Cam1 -->--> GPU --> CPU (frame processing, and data annotation )

```

More specifically -

```auto
Cam1 -->-->map-and-enqueue--> GPU --> CPU (frame processing, and data annotation ) --->dequeue-and-unmap

```

I am wondering whether action of mapping an input frame, for storage in a queue could be causing any conversion of format ?

THis is what I do -

1. Acquire a frame, map it for further access on GPU and CPU.

```auto
UniqueObj<Frame> frame(iFrameConsumer->acquireFrame());
        IFrame *iFrame = interface_cast<IFrame>(frame);

        if (!iFrame)
            break;

        // Get the Frame's Image.
        Image *image = iFrame->getImage();

        IArgusCaptureMetadata *iArgusCaptureMetadata = interface_cast<IArgusCaptureMetadata>(frame);
        if (!iArgusCaptureMetadata)
            ORIGINATE_ERROR("Failed to get IArgusCaptureMetadata interface.");
        CaptureMetadata *metadata = iArgusCaptureMetadata->getMetadata();
        ICaptureMetadata *iMetadata = interface_cast<ICaptureMetadata>(metadata);
        if (!iMetadata)
            ORIGINATE_ERROR("Failed to get ICaptureMetadata interface.");

        EGLStream::NV::IImageNativeBuffer *iImageNativeBuffer
              = interface_cast<EGLStream::NV::IImageNativeBuffer>(image);
        TEST_ERROR_RETURN(!iImageNativeBuffer, "Failed to create an IImageNativeBuffer");

        // aaFrameBuffer is a data struct which encapsulates, few pointers and a NvBuffer.
        aaFrameBuffer *framedata = new aaFrameBuffer;
               framedata->framefd = iImageNativeBuffer->createNvBuffer(ARGUSSIZE {m_pCamInfo->liveParams.inputVideoInfo.width, m_pCamInfo->liveParams.inputVideoInfo.height},
               NvBufferColorFormat_YUV420, NvBufferLayout_Pitch, &status);

		NvBufferGetParams(framedata->framefd, &(framedata->nvBufParams));
		framedata->fsizeY = framedata->nvBufParams.offset[1] + (framedata->nvBufParams.offset[2]-framedata->nvBufParams.offset[1])*2;
		framedata->fsizeU = framedata->nvBufParams.pitch[1] * framedata->nvBufParams.height[1] ;
		framedata->fsizeV = framedata->nvBufParams.pitch[2] * framedata->nvBufParams.height[2];

		m_pCamInfo->procInfo.pitchWidthY = framedata->nvBufParams.pitch[0];
		m_pCamInfo->procInfo.pitchWidthU = framedata->nvBufParams.pitch[1];
		m_pCamInfo->procInfo.pitchWidthV = framedata->nvBufParams.pitch[2];

		AACAM_CAPTURE_PRINT("4 Starting frame caputre %d \n",m_currentFrame);

          	framedata->dataY = (char *)mmap(NULL, framedata->fsizeY, PROT_READ | PROT_WRITE, MAP_SHARED, framedata->framefd, framedata->nvBufParams.offset[0]);
		framedata->dataU = (char *)mmap(NULL, framedata->fsizeU, PROT_READ | PROT_WRITE, MAP_SHARED, framedata->framefd, framedata->nvBufParams.offset[1]);
		framedata->dataV = (char *)mmap(NULL, framedata->fsizeV, PROT_READ | PROT_WRITE, MAP_SHARED, framedata->framefd, framedata->nvBufParams.offset[2]);

```

1. Frame from step 1, is put in Q.

2. GPU reads from the Q and processes it

3. CPU reads output of GPU and older frame from Q

4. After a delay of about 8 frames - a given frame is popped from Q , unmapped and destructed.

My question is whether mapping a frame fd could cause any conversion ?

Thanks,

---

<div class="post-metadata">

**Author:** ![DaneLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/danelll/32/14043_2.png) [@DaneLLL](https://forums.developer.nvidia.com/u/DaneLLL)\
**Post date:** [December 3, 2018, 1:43am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/15 "2018-12-03T01:43:32Z")

</div>

Hi,  
Please use NvBufferMemMap() instead of mmap().  
NvBufferMemMap() does not do any conversion.

---

<div class="post-metadata">

**Author:** ![dumbogeorge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dumbogeorge](https://forums.developer.nvidia.com/u/dumbogeorge)\
**Post date:** [December 3, 2018, 2:17am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/16 "2018-12-03T02:17:39Z")

</div>

Hi DaneLLL

Do you think that mmap could be the reason for some of v4l2convert.so calls I mentioned in #8 ?

Thanks

---

<div class="post-metadata">

**Author:** ![DaneLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/danelll/32/14043_2.png) [@DaneLLL](https://forums.developer.nvidia.com/u/DaneLLL)\
**Post date:** [December 3, 2018, 3:16am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/17 "2018-12-03T03:16:41Z")

</div>

Hi,  
I am not sure about mmap() and v4l2convert.so. But for NvBuffer, you should use APIs defined in nvbuf\_utils.h

```auto
/**
* This method must be used for hw memory cache sync for the CPU.
* @param[in] dmabuf_fd DMABUF FD of buffer.
* @param[in] plane video frame plane.
* @param[in] pVirtAddr Virtual Addres pointer of the mem mapped plane.
*
* @returns 0 for success, -1 for failure.
*/
int NvBufferMemSyncForCpu (int dmabuf_fd, unsigned int plane, void **pVirtAddr);

/**
* This method must be used for hw memory cache sync for device.
* @param[in] dmabuf_fd DMABUF FD of buffer.
* @param[in] plane video frame plane.
* @param[in] pVirtAddr Virtual Addres pointer of the mem mapped plane.
*
* @returns 0 for success, -1 for failure.
*/
int NvBufferMemSyncForDevice (int dmabuf_fd, unsigned int plane, void **pVirtAddr);

/**
* This method must be used for getting mem mapped virtual Address of the plane.
* @param[in] dmabuf_fd DMABUF FD of buffer.
* @param[in] plane video frame plane.
* @param[in] memflag NvBuffer memory flag.
* @param[in] pVirtAddr Virtual Addres pointer of the mem mapped plane.
*
* @returns 0 for success, -1 for failure.
*/
int NvBufferMemMap (int dmabuf_fd, unsigned int plane, NvBufferMemFlags memflag, void **pVirtAddr);

/**
* This method must be used to Unmap the mapped virtual Address of the plane.
* @param[in] dmabuf_fd DMABUF FD of buffer.
* @param[in] plane video frame plane.
* @param[in] pVirtAddr mem mapped Virtual Addres pointer of the plane.
*
* @returns 0 for success, -1 for failure.
*/
int NvBufferMemUnMap (int dmabuf_fd, unsigned int plane, void **pVirtAddr);

```

---

<div class="post-metadata">

**Author:** ![dumbogeorge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dumbogeorge](https://forums.developer.nvidia.com/u/dumbogeorge)\
**Post date:** [January 4, 2019, 10:24am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/18 "2019-01-04T10:24:12Z")

</div>

Hi DaneLLL

Would like to check about your answer #12

```auto
sensor -> ioctl(VIDIOC_DQBUF) -> captured frames in raw format -> VI/ISP -> frames in I420/NV12 format

```

Is there a way to avoid VIDIOC\_DQBUF call ? I see this happening with argus\_camera app too. Does each arrow here mean an read and write transaction to external memory (DRAM) ? That would seriously increase BW and degrade performance.

Is there a way for data to directly go to VI/ISP ? and not be routed via external memory ?

THanks

---

<div class="post-metadata">

**Author:** ![DaneLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/danelll/32/14043_2.png) [@DaneLLL](https://forums.developer.nvidia.com/u/DaneLLL)\
**Post date:** [January 7, 2019, 2:12am UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/19 "2019-01-07T02:12:02Z")

</div>

Hi,  
Argus frameworks is optimal and no extra memory copy. All operations are required for sensor frame capture. It has to take reasonable CPU/memory bandwidth.

> [@](#):
>
> Is there a way for data to directly go to VI/ISP ? and not be routed via external memory ?

So far we don’t have plan to support it.

---

<div class="post-metadata">

**Author:** ![nadeemm](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nadeemm/32/367020_2.png) [@nadeemm](https://forums.developer.nvidia.com/u/nadeemm)\
**Post date:** [October 18, 2021, 6:31pm UTC](https://forums.developer.nvidia.com/t/performance-optimization-help/67567/20 "2021-10-18T18:31:14Z")

</div>

Closed on October 18, 2021, 6:31pm UTC