IPP Capture - NvMedia Sample Plugin not working

Please provide the following info:
Hardware Platform: DRIVE AGX Xavier™ Developer Kit, DRIVE AGX Pegasus™ Developer Kit.
Software Version: Drive Software 10.0 (rev.1).
Host Machine Version: native Ubuntu 18.04
Image Sensor: Sekonix SF3324

Hi,

After I upgraded from Drive Software 9 to Drive Software 10.0, I am not able to use the IPP Raw (nvmipp_raw) with the sample plugin. Because the AutoExposure and Auto White Balance (AWB) algorithms are broken.
I am getting nvmedia errors of the type: failed to populateSensorAttributes. I attached the complete log and command to reproduce, also an image result.

In the next link, NVIDIA mentioned that this is a known issue. But my work currently depends on this and I am blocked. After some effort, I modified the sample plugin source to get the AE working, but the AWB is still broken. Is there a way for me to fix the AWB sample plugin?

Current output:

Complete command and log:

x11/nvmipp_raw -cf ./ddpx-a.conf -c SF3324-CSI-A --disable-egl --sampleplugin --aggregate 1 -v 2 --save-isp -n 10

nvmedia: Using params set: SF3324-CSI-A for capture
nvmedia: Using params set: SF3324-CSI-A for capture
The file save option has been selected.
Please note that this option is not guaranteed to save frames reliably without frame drops.
nvmedia isc: GetCameraPowerControlLevel: 936: Camera power control library: NVCCP
Max96712 Rev 2 detected!
MAX96712: Enable periodic AEQ on Link 0
MAX96705: Pre-emphasis set to 0xaa
Sensor AR0231 RCCB Rev7 detected!
nvmedia: Capture params:
Interface type: 0
Start X,Y: 0,0
nvmedia: resolution: 1920x1208
extra-lines: 28
interface-lanes: 2
nvmedia: ERROR: populateSensorAttributes: Failed to get min exp time sensor attribute
nvmedia: ERROR: populateSensorAttributes: Failed to get max exp time sensor attribute
nvmedia: ERROR: populateSensorAttributes: Failed to get min gain sensor attribute
nvmedia: ERROR: populateSensorAttributes: Failed to get max gain sensor attribute
nvmedia: ERROR: populateSensorAttributes: Failed to get min exp time sensor attribute
nvmedia: ERROR: populateSensorAttributes: Failed to get max exp time sensor attribute
nvmedia: ERROR: populateSensorAttributes: Failed to get min gain sensor attribute
nvmedia: ERROR: populateSensorAttributes: Failed to get max gain sensor attribute
nvmedia: Get IPP output thread is active, ippNum=1
Type "h" for a list of options
Saving frames...
nvmedia: [CameraId: 0 Frame: 9 CaptureTimestamp: 3833100110 CaptureTimeTick: 19451740373522] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 9] - CA DONE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 9] - ISC DONE
nvmedia: [CameraId: 0 Frame: 10 CaptureTimestamp: 3833133438 CaptureTimeTick: 19451741414582] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 10] - CA DONE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 10] - ISC DONE
nvmedia: [CameraId: 0 Frame: 11 CaptureTimestamp: 3833166743 CaptureTimeTick: 19451742456411] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 11] - CA DONE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 11] - ISC DONE
nvmedia: [CameraId: 0 Frame: 12 CaptureTimestamp: 3833200076 CaptureTimeTick: 19451743497472] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 12] - CA DONE
nvmedia: [CameraId: 0 Frame: 12] - ISC DONE
nvmedia: [CameraId: 0 Frame: 13 CaptureTimestamp: 3833233425 CaptureTimeTick: 19451744540069] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 13] - CA DONE
nvmedia: [CameraId: 0 Frame: 13] - ISC DONE
nvmedia: [CameraId: 0 Frame: 14 CaptureTimestamp: 3833266768 CaptureTimeTick: 19451745581897] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 14] - CA DONE
nvmedia: [CameraId: 0 Frame: 14] - ISC DONE
nvmedia: [CameraId: 0 Frame: 15 CaptureTimestamp: 3833300127 CaptureTimeTick: 19451746622993] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 15] - CA DONE
nvmedia: [CameraId: 0 Frame: 15] - ISC DONE
nvmedia: [CameraId: 0 Frame: 16 CaptureTimestamp: 3833333449 CaptureTimeTick: 19451747665592] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 16] - CA DONE
nvmedia: [CameraId: 0 Frame: 16] - ISC DONE
nvmedia: [CameraId: 0 Frame: 17 CaptureTimestamp: 3833366765 CaptureTimeTick: 19451748706615] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 17] - CA DONE
nvmedia: [CameraId: 0 Frame: 17] - ISC DONE
nvmedia: [CameraId: 0 Frame: 18 CaptureTimestamp: 3833400122 CaptureTimeTick: 19451749748443] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 18] - CA DONE
nvmedia: [CameraId: 0 Frame: 18] - ISC DONE
nvmedia: [CameraId: 0 Frame: 19 CaptureTimestamp: 3833433411 CaptureTimeTick: 19451750789505] - ICP CAPTURE
nvmedia: [CameraId: 0 Frame: 0] - ISP DONE
nvmedia: [CameraId: 0 Frame: 20 CaptureTimestamp: 3833466781 CaptureTimeTick: 19451751832102] - ICP CAPTURE

Hi @abraham.arias,

How did you get AE working? by calling NvMediaISCGetSensorAttributes() to get those attributes?
What do you mean “fix the AWB sample plugin”? Can you modify IPPPluginSimpleAWB() in sample_plugin.c for it?

Hi @VickNV.

No, I did not use NvMediaISCGetSensorAttributes. In order to make AE work I manually hardcoded the values as described below:
In populateSensorAttributes(NvMediaIPPPlugin *pluginHandle) in sample_plugin.c I hardcoded the following values:

minET[i] = 0.000057;
maxET[i] = 0.032856
minGain[i] = 0.8
maxGain[i] = 95.953125
fineET[i] // Not set, because behavior didn’t seem to change
stepET[i] // Not set, because behavior didn’t seem to change

This workaround allowed me to use AE. However, the AWB still produces the greenish image from above. There are a bunch of other associated problems, for example, I noticed that the IPPPluginSimpleAWB divides the computed gains by the input gains in NvMediaIPPPluginInput. These are sometimes zero which produces invalid gains.
The AWB algorithm seems to need valid gain ranges, but I’m not able to find them.

  1. Can you explain to me how to use NvMediaISCGetSensorAttributes in order to get valid sensor limits
  2. Is there an estimated date for a release with this fix?
  3. If this fix is waiting on other releases, would it be possible to receive the modified libraries?

Could you elaborate what are sometimes zero through your observation?

Basically, on the process function of the Sample Plugin. At the 16th frame or near, one incoming WB gain is zero. The AWB algorithm then makes a division by zero… and after that, the AWB produces absurd values. The sample plugin receives this through the following object:
NvMediaIPPPluginInput-> NvMediaISCWBGainControl -> gains.

This is happening in the ProcessExFunction. According to the docs, this function receives information from the previous component. This information is incorrect after the first frames.

This function is called at capture speed, one call per image. The plugin receives input information in the pluginInput structure and puts the results into the pluginOutput structure.

Does anyone have an idea of what is going on here with the ISP and ISC components at the IPP Capture? Can you please elaborate on how NVIDIA fixed this?

Hi @abraham.arias,

Because IPP will be depreccated in the next release (IPP will be lasted for only the release though), I would suggest you to move your efforts onto SIPL.

As I mentioned in below topic, you can peek at the SIPL camera sample to know how plugin will work in it.

Ok, thanks. I am checking the SIPL Camera Sample Application to make the switch but I encounter a problem.

The latest SDK Manager (1.1.0.6343) downloaded the SIPL Camera NvMedia Sample Application. I need as well the SIPL Reprocess NvMedia Sample Application but this second example is nowhere at the downloads.

Where can I find this?

Regards.

The docs are not for DRIVE Software 10.0 so you will see the samples/features in the upcoming release.

Oh, I see, thanks.

For many reasons, it would be crucial to know an estimate for this release. Weeks or months, maybe.

Regards.

Hi @abraham.arias,

We don’t disclose the schedule in the forum. If it may affect your product or schedule, please contact your nvidia representive.

Or if you still want us to help on the greenish problem on IPP which you saw, please provide the patch of our sample for our reproducing.