Getting histogram data in 5.2.0

I am currently implementing something requiring a histogram of the image. I am on 5.1.6, but the team is in the process of transitioning to 5.2. How do I query the histogram from the ISP under 5.2.0? So far I haven’t managed to decipher it from the documentation. nvmedia_isp.h no longer seems to exist. nvmeda_isp_stat.h is still there, but I don’t see a function to actually retrieve such inforomation. There is also NvSIPLControlAutoDef.hpp with relevant seeming struct-s, but again, I don’t see how to retrieve these. Could someone provide some pointers, please?

Please provide the following info (check/uncheck the boxes after clicking “+ Create Topic”):
Software Version
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.5.0.7774
other - none

Host Machine Version
native Ubuntu 18.04
other

Hi @kornel.lehocz.ext ,

Please check if Process() in ~/nvidia/nvidia_sdk/DRIVE_OS_5.2.0_SDK_Linux_OS_DDPX/DRIVEOS/drive-t186ref-linux/samples/nvmedia/nvsipl/test/camera/CSamplePlugin.hpp helps. Thanks.

Hi @VickNV ,
Thanks for the quick response! I decided to implement a temporarily solution for 5.1.6, which we are still using. The API there for processing an image (and getting the histogram) seemed fairly simple. However, I ran into the following issue: the ISP configuration doesn’t seem to be used, thus everything is purple. I found the function NvMediaISPSettingsLoadConfig, but it’s not clear what this binary blob format is. I have the config in a text file. (and potentially a nito file for 5.2.0) Can you advise how I can load the settings (ideally without having to manually recreate the contents with API calls that are deprecated in 5.2.0) ?

Alternatively, is there any way to retrieve an ISP object (for the Image Signal Processing API) from the INvSIPLClient API? Or a way to get a histogram there, that’s available under 5.1.6?

I tried setting the calibration matrix from the isp file using the NvMedia signal processing API, but the image is still very purple (but it can be made out and there are also colours). Any ideas why?

Another problem I have is that the histogram is basically empty - everything is in the topmost bin according to it. I am enabling it and all NvMedia function calls succeed.
I use the following settings (expecting a simple histogram):
controls.enable = true;
controls.offset = 0.0f; //Holds offset to be applied to input data prior to bin mapping
controls.knees[0] = 32 - 1;
controls.knees[1] = 232 - 1;
controls.knees[2] = 3
32 - 1;
controls.knees[3] = 432 - 1;
controls.knees[4] = 5
32 - 1;
controls.knees[5] = 632 - 1;
controls.knees[6] = 7
32 - 1;
controls.knees[7] = 8*32 - 1;

controls.ranges[0] = 5;   //Holds \(log_2\) range of the pixel values to be considered for each zone
controls.ranges[1] = 5;   //
controls.ranges[2] = 5;   //
controls.ranges[3] = 5;   //
controls.ranges[4] = 5;   //
controls.ranges[5] = 5;   //
controls.ranges[6] = 5;   //
controls.ranges[7] = 5;   //

What exactly are these ranges? Am I setting it correctly for a “normal” histogram?
Is there documentation about this somewhere? (other than the very brief reference guide, which doesn’t really explain)

Please elaborate what you are doing. Can we observe the issue with any sample application?

NVIDIA DRIVE OS Linux SDK API Reference: NvMediaISPHistogramStats Struct Reference | NVIDIA Docs is the documentation about this.

I’m trying to get the histogram of the image processed by the Tegra hardware ISP, under 5.1.6 now. I’m using the NvMedia image signal processing API (nvmedia_isp.h). I cannot find any relevant sample, could you point me to one please?

The image is fine in another component, where the SIPL client API is used, but I don’t see any way to retrieve the histogram there under 5.1.6. If it was possible to do it there, that would be even better, as that’s what our system is currently using. Do you maybe have any pointers for that?

My current plan is to only get the raw image using SIPL, and do the raw processing using the nvmedia_isp API, but apparently the ISP is not configured correctly in this case - our camera_overrides.isp settings aren’t being used. I tried setting the colour calibration matrix from the isp file (using an API call in nvmedia_isp.h), and it does make a slight difference, but the image is still very purple, and I’m unable to figure out why. Is it maybe possible to get a handle of an ISP object (that’s correctly configured) from the SIPL consumer that I can use with the nvmedia_isp API?

Update: the green channel seems to be completely missing. I am currently converting from the NvMedia raw surface that the SIPL consumer gets to the same NV12 surface it processes to (currently still processing twice). I’ve tried only processing a crop region, so I see both on the same image. The SIPL part of the image is fine. The image from the NvMedia ISP API is there, but green is missing. I don’t see any way to set the sensor layout (RGGB) in nvmedia_isp.h. Any tips?

To avoid the topic is cluttered with information for both DRIVE OS 5.1.6.1 and 5.2.0. Please make sure you’re using different topics for different releases. May I know on which release you discuss problems in this topic? Thanks

I’ve gotten it to work under 5.2.0 based on the sample. Thanks for the pointer!

What is the correct way to set the ranges for the processed RGB histogram? This is unfortunately not clear from the sample, as the Process function receives the ranges as input parameters for histogram 0. For histogram 1 (which is of the processed RGB image, an I correct?) I need to specify the ranges. Setting it up like in the code snippet I posted above doesn’t work. I’ve experimented with it, and have gotten a plausible looking histogram by setting the ranges to 17, but the needed range seems to change based on exposure time (?).

Btw. does this auto control sample replicate the behaviour of the default Nvidia auto control plugin? Or does the default one do more sophisticated exposure control or white balancing? Ie. am I loosing any functionality by using and extending this instead? (which appears to be the only way to access the histogram)

NV_PLUGIN is different from the sample plugin.

Please see if information in “Registering a User Defined Auto Control Plugin” doc section helps.

Please take a peek at the following ImageMetaData reference of upcoming 5.2.6 release. It will support histogram.

https://docs.nvidia.com/drive/drive-os-5.2.6.0L/drive-os/DRIVE_OS_Linux_SDK_Development_Guide/baggage/structnvsipl_1_1INvSIPLClient_1_1ImageMetaData.html

Please take a peek at the following ImageMetaData reference of upcoming 5.2.6 release. It will support histogram.

That sounds great! Do you have an ETA for 5.2.6?

We don’t share release date information in the forum because of its tentative nature.
I would say it’s right around the corner and you can always check if your nvidia representative can share it.
Sorry for any inconvenience.