Dedicated Stereo vision IP

Dear Nvidia support team,

during my investigation of what is inside the Tegra Xavier SoC, I found information about a dedicated Stereo Vision and Optical Flow engine:
https://en.wikichip.org/wiki/nvidia/tegra/xavier#Stereo_.26_Optical_Flow_Engine
or
https://images.anandtech.com/doci/13584/1.12_NVIDIA_XavierHotchips2018Final_8144.jpg

But this engine is not mentioned at the official Nvidia webpages:
https://developer.nvidia.com/embedded/jetson-agx-xavier

Please, is there really a dedicated hardware accelerator for Stereo Vision in Xavier SoC? Or GPU or the programmable Vision processor must be used to achieve a reasonable FPS?

Thank you!

Hi JanTV1, upcoming support for hardware-accelerated stereo disparity depth mapping will be included for Xavier in the next JetPack release - please stay tuned. For now, you can use GPU to process stereo disparity.

Hi Dusty_nv,

thank you for your answer!
Based on my understanding, the JetPack is a software package. My question is maybe more HW related.
We are in a decision process of selecting a SoC vendor for a next project and the Stereo IP block may be critical as the other HW resources (GPU/VP) would be used for other compute intensive tasks.

I understand if this information isn’t public today. Do you recommend to contact a sales person for these questions?

Thank you!

JetPack includes the software libraries that interface to the hardware, like the GPU, DLA’s, video encoder/decoder, CSI/ISP engine, ect. In the case of the Vision Accelerator hardware on Xavier, a library will be added to the next JetPack release which interfaces to it as well, and includes support for hardware-accelerated stereo disparity depth mapping.

I’m sorry, but it still doesn’t answer my question. Please, let forget about the software libraries or drivers. I understand that point. My question is hardware related.

Based on the picture I sent before:
https://images.anandtech.com/doci/13584/1.12_NVIDIA_XavierHotchips2018Final_8144.jpg
Those are two different HW accelerators:

  • Vision Accelerator (sometimes referred as PSA) with mentioned performance 1.7 TOPS.
  • Stereo and Optical flow engine (sometimes referred as SOFE) with mentioned performance 2x3.1TOPS.

In my question, I don’t ask about the “PSA - Vision Accelerator” (although I believe Stereo vision algorithm can be implemented with this accelerator as well and we can call it “hardware accelerated”).

I’m asking about the “SOFE - Stereo and Optical flow engine”. Based on that picture, my understanding is that the SOFE is some hardwired/fixed functional accelerator dedicated for Stereo vision and Optical flow ONLY. So PSA and SOFE are two different HW IPs.
So, is there really IP like that in Xavier AGX SoC?

Thank you!

These have not been explicitly advertised as separate HW units in Jetson AGX Xavier, and they would all fall under the guise of vision accelerators which are not exposed directly to the user but through the software APIs. In the upcoming software libraries, the appropriate hardware engine would be used automatically depending on the algorithm, so you wouldn’t need to worry about it.

Thank you for your answer, Dusty_nv. I understand.
Please, I have an additional question regarding the software API.
Which software library will be used to program these vision accelerators? I’m wondering if users can use some lower-level API (something like CUDA) to optimize their algorithms or there will be a higher level API only (like OpenVX)?
Can I theoretically implement an arbitrary algorithm for these vision accelerators (for example if I want to use my own stereo vision algorithm and not the predefined one)?

I’m sorry for too many questions, I just would like to know if the new accelerators can be used as a replacement of GPUs (or to offload GPU) as a fully programmable architecture or it will be dedicated for a few tasks only.

Thank you!

Hi,

Sorry that we cannot disclosure any detail about our upcoming libraries.
Please pay attention to our announcement for the first-hand information.

For deep learning use case, you can offload GPU job to the DLA.
You can find more information about DLA here: http://nvdla.org/

Thanks.

Hi AastaLLL,

I understand.
Thank you for the explanation.