Video Image Compositor

Hi,
The sample for fisheye input is in

/opt/nvidia/vpi/samples/11-fisheye

You may get EglImage from Argus and call vpiImageWrapEglImage() to get VPIImage.

DaneLLL,

How do I add a feature request to the JetPack?

It would really make sense to utilize the VIC accellerator of the Xavier for duties of image convertion.
Featuers to support

  • TNR, nvarguscamerasrc ?
  • Lens Distortion Correction (dewarping), nvarguscamerasrc or nvvidconv ?
  • Cropping, nvarguscamerasrc or nvvidconv ?
  • Warping, nvvidconvert?
  • Etc.

The Xavier is a very powerful SoC, and it would make sense tu utilize it’s full power.

Hi,
Have checked with internal teams and would like to clarify the VPI functions are done on hardware VIC engine.
https://docs.nvidia.com/vpi/algo_ldc.html

The samples are in

/usr/src/jetson_multimedia_api/
/opt/nvidia/vpi/samples/

You would need to look at the samples and do integration. Suggest check if your source can be launched through Argus first.

Using the VPI library seems nice, but I can’t get the performance out of it.

E.g running the 10-perspwarp example, instrumented with timing measurements with an imput video of 4032x3040 resolution

Frame: 563, warp-time: 90.340721 ms
Frame: 564, warp-time: 102.452217 ms
Frame: 565, warp-time: 104.776291 ms
Frame: 566, warp-time: 107.715652 ms
Frame: 567, warp-time: 107.427452 ms
Frame: 568, warp-time: 103.814690 ms
Frame: 569, warp-time: 102.359993 ms

The instrumented code

    float elapsedWarp;
    CHECK_STATUS(vpiEventElapsedTime(ev1, ev2, &elapsedWarp));

The Tegrastats while running

RAM 4644/7763MB (lfb 267x4MB) CPU [35%@1420,45%@1420,30%@1420,29%@1420,24%@1420,52%@1420] EMC_FREQ 18%@1600 GR3D_FREQ 0%@408 APE 150 MTS fg 0% bg 5% AO@41C GPU@41.5C PMIC@100C AUX@42C CPU@43.5C thermal@42.15C VDD_IN 6410/5074 VDD_CPU_GPU_CV 1674/912 VDD_SOC 1467/1233
RAM 4644/7763MB (lfb 267x4MB) CPU [35%@1420,49%@1420,53%@1420,52%@1420,35%@1420,59%@1420] EMC_FREQ 19%@1600 GR3D_FREQ 34%@408 APE 150 MTS fg 0% bg 7% AO@41.5C GPU@41.5C PMIC@100C AUX@42C CPU@43.5C thermal@42.3C VDD_IN 6900/5076 VDD_CPU_GPU_CV 2123/913 VDD_SOC 1508/1234
RAM 4643/7763MB (lfb 267x4MB) CPU [45%@1420,41%@1420,52%@1420,33%@1420,52%@1420,72%@1420] EMC_FREQ 19%@1600 GR3D_FREQ 22%@306 APE 150 MTS fg 0% bg 6% AO@41.5C GPU@41.5C PMIC@100C AUX@42C CPU@43.5C thermal@42.3C VDD_IN 6818/5077 VDD_CPU_GPU_CV 2082/914 VDD_SOC 1467/1234
RAM 4643/7763MB (lfb 267x4MB) CPU [39%@1420,65%@1420,28%@1420,52%@1420,49%@1420,15%@1420] EMC_FREQ 18%@1600 GR3D_FREQ 35%@306 APE 150 MTS fg 0% bg 6% AO@41.5C GPU@41.5C PMIC@100C AUX@42.5C CPU@43.5C thermal@42.45C VDD_IN 6573/5079 VDD_CPU_GPU_CV 1878/914 VDD_SOC 1426/1234
RAM 4644/7763MB (lfb 267x4MB) CPU [37%@1420,29%@1420,16%@1420,68%@1420,39%@1420,23%@1420] EMC_FREQ 16%@1600 GR3D_FREQ 11%@306 APE 150 MTS fg 0% bg 7% AO@41.5C GPU@42C PMIC@100C AUX@42.5C CPU@43.5C thermal@42.8C VDD_IN 6165/5080 VDD_CPU_GPU_CV 1592/915 VDD_SOC 1426/1234
RAM 4643/7763MB (lfb 267x4MB) CPU [41%@1420,25%@1420,31%@1420,30%@1420,24%@1420,58%@1420] EMC_FREQ 16%@1600 GR3D_FREQ 27%@306 APE 150 MTS fg 0% bg 6% AO@42C GPU@42C PMIC@100C AUX@42.5C CPU@44C thermal@42.65C VDD_IN 6288/5081 VDD_CPU_GPU_CV 1674/916 VDD_SOC 1426/1234
RAM 4643/7763MB (lfb 267x4MB) CPU [45%@1420,31%@1420,29%@1420,19%@1420,59%@1420,40%@1420] EMC_FREQ 17%@1600 GR3D_FREQ 53%@306 APE 150 MTS fg 0% bg 5% AO@42C GPU@42C PMIC@100C AUX@42.5C CPU@44C thermal@42.65C VDD_IN 6614/5082 VDD_CPU_GPU_CV 1878/916 VDD_SOC 1467/1235
RAM 4643/7763MB (lfb 267x4MB) CPU [42%@1420,25%@1420,69%@1420,44%@1420,40%@1420,18%@1420] EMC_FREQ 17%@1600 GR3D_FREQ 12%@306 APE 150 MTS fg 0% bg 5% AO@42C GPU@42C PMIC@100C AUX@42.5C CPU@44C thermal@42.8C VDD_IN 6492/5083 VDD_CPU_GPU_CV 1837/917 VDD_SOC 1426/1235
RAM 4643/7763MB (lfb 267x4MB) CPU [52%@1420,32%@1420,30%@1420,66%@1420,34%@1420,37%@1420] EMC_FREQ 17%@1600 GR3D_FREQ 14%@306 APE 150 MTS fg 0% bg 5% AO@42C GPU@42C PMIC@100C AUX@43C CPU@44C thermal@42.95C VDD_IN 6573/5085 VDD_CPU_GPU_CV 1919/918 VDD_SOC 1467/1235
RAM 4643/7763MB (lfb 267x4MB) CPU [41%@1420,37%@1420,19%@1420,55%@1420,58%@1420,16%@1420] EMC_FREQ 16%@1600 GR3D_FREQ 15%@306 APE 150 MTS fg 0% bg 8% AO@42C GPU@42.5C PMIC@100C AUX@42.5C CPU@44.5C thermal@43.3C VDD_IN 6410/5086 VDD_CPU_GPU_CV 1796/919 VDD_SOC 1426/1235

Hi,
We would need your help to share a test sample so that we can reproduce the performance issue and do investigation.

Hi,

Here’s an update on the 10-perspwarp example

Withe the dashcam.mp4 example file, that is 1280 × 720 resolution, the timing is about 9-10ms.

Whereas the performance for a 1920*1080 shuld be about 2.2ms. Even considering the clock diffferenec between AGX xavier adn Xavier NX doubling the time to 5m. The corresponding time for the 1280x720 should be about 2 ms.

main.cpp (10.5 KB)

Hi,

We are checking this with our internal team.
Will update more information with you later.

Thanks.

Hi,

Sorry to keep you waiting.

In our VPI API, it uses VIC hardware for perspective warping.
This information will be updated in our future document.

As a result, it’s essential to maximize the VIC clock rate to reach optimal performance.

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

Then maximize VIC with this comment.

With this, we can get 2.2ms on Xaiver and 3.7ms on XaiverNX, which is similar to the document.

Thanks.

Thanks,

So the ImageRemap feature uses the VIC, not the PVA as the current documentation says?

Hi,

More precisely, function vpiSubmitPerspectiveImageWarp use VIC rather than PVA.
This information will be added to our future document.

Sorry for any inconvenience.
Thanks.

Will the VIC be used for the vpiSubmitImageRemap feature as well?

/Fredrik

Hi,

Let me check this with our internal team first.

Thanks.

Hi,

Just confirmed.
vpiSubmitImageRemap is also running on the VIC.

Thanks.