How to synchronize NvBufferTransform reading from an Argus IBufferOutputStream with SYNC_TYPE_EGL_SYNC?

[cory.bloyd]

SYNC_TYPE_EGL_SYNC looks great for OpenGL and CUDA based pipelines. But, I also need to run my captured images through NvBufferTransform (preferably NvBufferTransformAsync). How do I fence the nvbuf_utils API to synchronize it’s access to the NvBuffers that are written to by the Argus stream?

[DaneLLL]

Hi,
Not sure understand the usecase. Could you share your current function calls and where you would like to put NvBufferTransform()?

[cory.bloyd]

I already have a ring of NvBuffers that Argus is writing to. Those buffers are consumed by 3 different code paths: CUDA to leverage it’s flexibility, OpenGL to leverage the rasterization hardware, and nvbuf_utils to leverage the VIC/ISP.

I’m currently calling acquireBuffer on an IBufferOutputStream that is configured with SYNC_TYPE_NONE. I’d like to switch to SYNC_TYPE_EGL_SYNC to cut down on the latency between steps on the device side. With CUDA and GL it is clear how to use the EglSync object to fence the GPU operations on the buffer and everything should be wonderful there.

But, if I called acquireBuffer, then immediately call NvBufferTransform in the SYNC_TYPE_EGL_SYNC situation I would assume the Transform would erroneously execute before the buffer is ready. It’s not clear how to use an Argus IEGLSync object to fence NvBufferTransformAsync the same way that I would use it to fence device-side operation in OpenGL. The only option I see is to stall the host-side CPU before launching NvBufferTransformAsync --thus introducing a pipeline bubble.

Similarly, it is clear how to use NvBufferSyncObj to coordinate multiple calls to NvBufferTransformAsync. But, it is not clear how to use it to fence GL or CUDA.

I’m hoping there is a better solution than “Stall the CPU and deal with the device pipeline bubbles whenever you move images between nvbuff_utils and any other API.”

Is there one?

[DaneLLL]

Hi,
Please run sudo nvpmodel -m 0, sudo jetson_clocks and set VIC engin at max clock:

Nvvideoconvert issue, nvvideoconvert in DS4 is better than Ds5?

Hi, Please excute the steps to run VIC in max clock and try again: Disable runtime suspend of VIC $ echo on > /sys/devices/13e10000.host1x/15340000.vic/power/control Set userspace governor $ echo userspace > /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/governor Set max_freq $ cat /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/available_frequencies $ echo [max_freq_val] > /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/max_freq Set targ…

See if running the system at max performance helps.

[cory.bloyd]

@DaneLLL I’m sorry. I don’t see how this is related. Did you reply to the wrong thread?

[DaneLLL]

Hi,
NvBufferTransform() and NvBufferTransformAsync() use hardware VIC engine. Would like to know if it helps by running the engine at max clock.

[cory.bloyd]

Ah. Sorry. My question is not about performance; it is about correctness. Am I right that there is no “defined behavior” method to fence NvBufferTransformAsync with an IEGLSync object other than blocking the CPU on the IEGLSync before starting the BufferTransform?

[cory.bloyd]

Here. I’ve modified tegra_multimedia_api/argus/samples/eglImage/main.cpp to illustrate my questions.
The sample app is 500 lines of code so, I’ve marked all of my changes with

/********************
*********************
* Giant Comment Blocks Like This One!
*********************
*********************/

AsyncAcquireBufferExample.cpp (19.1 KB)

[carolyuu]

Hi cory.bloyd,

Using your code, but build fail:
Test on r32.4.3/Xavier

[ 72%] Building CXX object samples/eglImage/CMakeFiles/argus_eglimage.dir/main.cpp.o
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp: In member function ‘virtual bool ArgusSamples::EGLImageRenderingThread::threadExecute()’:
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:221:24: error: ‘PretendThereIsAFunctionToRetrieveTheNativeBuffersFileDescriptor’ was not declared in this scope
         int bufferFD = PretendThereIsAFunctionToRetrieveTheNativeBuffersFileDescriptor(buffer);
                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:223:9: error: ‘NvBufferCreateParams’ was not declared in this scope
         NvBufferCreateParams params;
         ^~~~~~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:224:9: error: ‘params’ was not declared in this scope
         params.width       = config.ARGBWidth;
         ^~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:224:30: error: ‘config’ was not declared in this scope
         params.width       = config.ARGBWidth;
                              ^~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:224:30: note: suggested alternative: ‘confstr’
         params.width       = config.ARGBWidth;
                              ^~~~~~
                              confstr
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:226:30: error: ‘NvBufferPayload_SurfArray’ was not declared in this scope
         params.payloadType = NvBufferPayload_SurfArray;
                              ^~~~~~~~~~~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:227:30: error: ‘NvBufferLayout_Pitch’ was not declared in this scope
         params.layout      = NvBufferLayout_Pitch;
                              ^~~~~~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:228:30: error: ‘NvBufferColorFormat_ABGR32’ was not declared in this scope
         params.colorFormat = NvBufferColorFormat_ABGR32;
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:229:30: error: ‘NvBufferTag_VIDEO_CONVERT’ was not declared in this scope
         params.nvbuf_tag   = NvBufferTag_VIDEO_CONVERT;
                              ^~~~~~~~~~~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:231:9: error: ‘NvBufferCreateEx’ was not declared in this scope
         NvBufferCreateEx(&destFD, &params);
         ^~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:231:9: note: suggested alternative: ‘glBufferData’
         NvBufferCreateEx(&destFD, &params);
         ^~~~~~~~~~~~~~~~
         glBufferData
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:232:33: error: ‘NvEGLImageFromFd’ was not declared in this scope
         EGLImageKHR egl_image = NvEGLImageFromFd(g_display.get(), destFD);
                                 ^~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:232:33: note: suggested alternative: ‘IEGLImageSource’
         EGLImageKHR egl_image = NvEGLImageFromFd(g_display.get(), destFD);
                                 ^~~~~~~~~~~~~~~~
                                 IEGLImageSource
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:234:9: error: ‘NvBufferTransformParams’ was not declared in this scope
         NvBufferTransformParams transform_params = {0};
         ^~~~~~~~~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:235:9: error: ‘transform_params’ was not declared in this scope
         transform_params.transform_flag = NVBUFFER_TRANSFORM_FILTER;
         ^~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:235:9: note: suggested alternative: ‘random_data’
         transform_params.transform_flag = NVBUFFER_TRANSFORM_FILTER;
         ^~~~~~~~~~~~~~~~
         random_data
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:235:43: error: ‘NVBUFFER_TRANSFORM_FILTER’ was not declared in this scope
         transform_params.transform_flag = NVBUFFER_TRANSFORM_FILTER;
                                           ^~~~~~~~~~~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:236:45: error: ‘NvBufferTransform_Filter_Smart’ was not declared in this scope
         transform_params.transform_filter = NvBufferTransform_Filter_Smart;
                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:238:9: error: ‘NvBufferSyncObj’ was not declared in this scope
         NvBufferSyncObj bufferSyncObj = PretendTheresAFunctionToMakeTheSyncObject();
         ^~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:254:44: error: ‘acquireSync’ was not declared in this scope
         eglClientWaitSync(g_display.get(), acquireSync, EGL_SYNC_FLUSH_COMMANDS_BIT, EGL_FOREVER);
                                            ^~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:259:70: error: ‘bufferSyncObj’ was not declared in this scope
         NvBufferTransformAsync(bufferFD, destFD, &transform_params, &bufferSyncObj);
                                                                      ^~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:259:9: error: ‘NvBufferTransformAsync’ was not declared in this scope
         NvBufferTransformAsync(bufferFD, destFD, &transform_params, &bufferSyncObj);
         ^~~~~~~~~~~~~~~~~~~~~~
/usr/src/jetson_multimedia_api/argus/samples/eglImage/main.cpp:276:9: error: ‘NvBufferSyncObjWait’ was not declared in this scope
         NvBufferSyncObjWait(&bufferSyncObj.outsyncobj, 0xFFFFFFFF);
         ^~~~~~~~~~~~~~~~~~~
samples/eglImage/CMakeFiles/argus_eglimage.dir/build.make:62: recipe for target 'samples/eglImage/CMakeFiles/argus_eglimage.dir/main.cpp.o' failed
make[2]: *** [samples/eglImage/CMakeFiles/argus_eglimage.dir/main.cpp.o] Error 1
CMakeFiles/Makefile2:926: recipe for target 'samples/eglImage/CMakeFiles/argus_eglimage.dir/all' failed
make[1]: *** [samples/eglImage/CMakeFiles/argus_eglimage.dir/all] Error 2
Makefile:151: recipe for target 'all' failed
make: *** [all] Error 2

[cory.bloyd]

My code is just an illustration. Additional code that would be slow to write, necessary to make it run, but not necessary to illustrate the problem is not implemented. Instead, I just pretend there is a function called PretendThereIsAFunctionToDoX();

[DaneLLL]

Hi,
It is more like a feature request and we would need your help to share a sample that we can compile and run to reproduce your observation and see if we can handle the case in future release(s).

In current release, we would suggest synchronize multiple sources with timestamp information. The timestamping mechanism is done in kernel and very accurate. You can check the timestamps to synchronize the sources.

[DaneLLL]

Hi,
We have Argus samples which are tested/verified in each release. There is no sample that demonstrates this case, so it is not tested and may have potential issue(s). Might be potential issue(s) in using OPenGL + NvBuffer APIs. Would be great if you can share a sample that we can build/run to observe the CPU gets stalled.

For more information, looks like it is based on

/usr/src/jetson_multimedia_api/argus/samples/openglBox

And add NvBuffer APIs?

[cory.bloyd]

Thanks!
I was pretty sure the API did not support my case. Just needed confirmation.

We are creating an image processing pipeline with many steps using a variety of HW APIs. And, it is a major goal to keep that pipeline tightly packed on the GPU and fenced by other HW accelerators rather than start-and-stop each step on the CPU. Thus the focus on CPU-async interfaces.

I’ll take a look at openglBox and either make a runnable sample out of that or flesh out my edit of tegra_multimedia_api/argus/samples/eglImage