I am currently implementing something requiring a histogram of the image. I am on 5.1.6, but the team is in the process of transitioning to 5.2. How do I query the histogram from the ISP under 5.2.0? So far I haven’t managed to decipher it from the documentation. nvmedia_isp.h no longer seems to exist. nvmeda_isp_stat.h is still there, but I don’t see a function to actually retrieve such inforomation. There is also NvSIPLControlAutoDef.hpp with relevant seeming struct-s, but again, I don’t see how to retrieve these. Could someone provide some pointers, please?
Please provide the following info (check/uncheck the boxes after clicking “+ Create Topic”): Software Version
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other
Please check if Process() in ~/nvidia/nvidia_sdk/DRIVE_OS_5.2.0_SDK_Linux_OS_DDPX/DRIVEOS/drive-t186ref-linux/samples/nvmedia/nvsipl/test/camera/CSamplePlugin.hpp helps. Thanks.
Hi @VickNV ,
Thanks for the quick response! I decided to implement a temporarily solution for 5.1.6, which we are still using. The API there for processing an image (and getting the histogram) seemed fairly simple. However, I ran into the following issue: the ISP configuration doesn’t seem to be used, thus everything is purple. I found the function NvMediaISPSettingsLoadConfig, but it’s not clear what this binary blob format is. I have the config in a text file. (and potentially a nito file for 5.2.0) Can you advise how I can load the settings (ideally without having to manually recreate the contents with API calls that are deprecated in 5.2.0) ?
Alternatively, is there any way to retrieve an ISP object (for the Image Signal Processing API) from the INvSIPLClient API? Or a way to get a histogram there, that’s available under 5.1.6?
I tried setting the calibration matrix from the isp file using the NvMedia signal processing API, but the image is still very purple (but it can be made out and there are also colours). Any ideas why?
Another problem I have is that the histogram is basically empty - everything is in the topmost bin according to it. I am enabling it and all NvMedia function calls succeed.
I use the following settings (expecting a simple histogram):
controls.enable = true;
controls.offset = 0.0f; //Holds offset to be applied to input data prior to bin mapping
controls.knees[0] = 32 - 1;
controls.knees[1] = 232 - 1;
controls.knees[2] = 332 - 1;
controls.knees[3] = 432 - 1;
controls.knees[4] = 532 - 1;
controls.knees[5] = 632 - 1;
controls.knees[6] = 732 - 1;
controls.knees[7] = 8*32 - 1;
controls.ranges[0] = 5; //Holds \(log_2\) range of the pixel values to be considered for each zone
controls.ranges[1] = 5; //
controls.ranges[2] = 5; //
controls.ranges[3] = 5; //
controls.ranges[4] = 5; //
controls.ranges[5] = 5; //
controls.ranges[6] = 5; //
controls.ranges[7] = 5; //
What exactly are these ranges? Am I setting it correctly for a “normal” histogram?
Is there documentation about this somewhere? (other than the very brief reference guide, which doesn’t really explain)
I’m trying to get the histogram of the image processed by the Tegra hardware ISP, under 5.1.6 now. I’m using the NvMedia image signal processing API (nvmedia_isp.h). I cannot find any relevant sample, could you point me to one please?
The image is fine in another component, where the SIPL client API is used, but I don’t see any way to retrieve the histogram there under 5.1.6. If it was possible to do it there, that would be even better, as that’s what our system is currently using. Do you maybe have any pointers for that?
My current plan is to only get the raw image using SIPL, and do the raw processing using the nvmedia_isp API, but apparently the ISP is not configured correctly in this case - our camera_overrides.isp settings aren’t being used. I tried setting the colour calibration matrix from the isp file (using an API call in nvmedia_isp.h), and it does make a slight difference, but the image is still very purple, and I’m unable to figure out why. Is it maybe possible to get a handle of an ISP object (that’s correctly configured) from the SIPL consumer that I can use with the nvmedia_isp API?
Update: the green channel seems to be completely missing. I am currently converting from the NvMedia raw surface that the SIPL consumer gets to the same NV12 surface it processes to (currently still processing twice). I’ve tried only processing a crop region, so I see both on the same image. The SIPL part of the image is fine. The image from the NvMedia ISP API is there, but green is missing. I don’t see any way to set the sensor layout (RGGB) in nvmedia_isp.h. Any tips?
To avoid the topic is cluttered with information for both DRIVE OS 5.1.6.1 and 5.2.0. Please make sure you’re using different topics for different releases. May I know on which release you discuss problems in this topic? Thanks
I’ve gotten it to work under 5.2.0 based on the sample. Thanks for the pointer!
What is the correct way to set the ranges for the processed RGB histogram? This is unfortunately not clear from the sample, as the Process function receives the ranges as input parameters for histogram 0. For histogram 1 (which is of the processed RGB image, an I correct?) I need to specify the ranges. Setting it up like in the code snippet I posted above doesn’t work. I’ve experimented with it, and have gotten a plausible looking histogram by setting the ranges to 17, but the needed range seems to change based on exposure time (?).
Btw. does this auto control sample replicate the behaviour of the default Nvidia auto control plugin? Or does the default one do more sophisticated exposure control or white balancing? Ie. am I loosing any functionality by using and extending this instead? (which appears to be the only way to access the histogram)
We don’t share release date information in the forum because of its tentative nature.
I would say it’s right around the corner and you can always check if your nvidia representative can share it.
Sorry for any inconvenience.
I’m still having trouble with interpreting the histogram and/or setting the ranges correctly.
It appears that histData[1], which I assumed to be of the processed image, changes significantly as I change the amount of light in our camera lab. Note that the processed image remains more or less the same - the exposure control adjusts to the lighting. The sensor gain and digital gain settings are both held constant. (btw. how do these interact with the histogram?)
(Btw. histData[0] does not change significantly under the same scenario - the changed exposure time results in more or less the same image and the histogram there is as expected. But it is the histogram of the processed image that I’d need, not of the raw data.)
How do I get the correct histogram of the processed image?
How does simulation mode (ie. feeding SIPL with a raw video) interact with the histogram? Are exposure / gain settings ignored?
Could someone from Nvidia maybe elaborate on some of these questions?
Would it be possible to have more documentation on these topics?
The auto control sample uses histogram 0 - this is of the raw, unprocessed image as I understand. To request histogram 1 (which I believe is of the processed RGB image), I set the following in ControlAuto::Process():
int range = 12;
outParams.newStatsSetting.valid = true;
outParams.newStatsSetting.hist1.enable = true;
outParams.newStatsSetting.hist1.knees[0] = 32 - 1;
outParams.newStatsSetting.hist1.knees[1] = 2*32 - 1;
outParams.newStatsSetting.hist1.knees[2] = 3*32 - 1;
outParams.newStatsSetting.hist1.knees[3] = 4*32 - 1;
outParams.newStatsSetting.hist1.knees[4] = 5*32 - 1;
outParams.newStatsSetting.hist1.knees[5] = 6*32 - 1;
outParams.newStatsSetting.hist1.knees[6] = 7*32 - 1;
outParams.newStatsSetting.hist1.knees[7] = 8*32 - 1;
outParams.newStatsSetting.hist1.ranges[0] = range; //Holds \(log_2\) range of the pixel values to be considered for each zone
outParams.newStatsSetting.hist1.ranges[1] = range; //
outParams.newStatsSetting.hist1.ranges[2] = range; //
outParams.newStatsSetting.hist1.ranges[3] = range; //
outParams.newStatsSetting.hist1.ranges[4] = range; //
outParams.newStatsSetting.hist1.ranges[5] = range; //
outParams.newStatsSetting.hist1.ranges[6] = range; //
outParams.newStatsSetting.hist1.ranges[7] = range; //
outParams.newStatsSetting.hist1.rectangularMask.x0 = 36;
outParams.newStatsSetting.hist1.rectangularMask.y0 = 148;
outParams.newStatsSetting.hist1.rectangularMask.x1 = 1900;
outParams.newStatsSetting.hist1.rectangularMask.y1 = 1068;
What I am expecting (hoping) to see is the histogram of the processed RGB image in inParams.statsInfo.histData[1] in the next frame. What I am getting is a histogram that changes with with the amount of light - eg. if I double the amount of light, the histogram scales up horizontally, while the processed image doesn’t actually change (only insignificantly), because the auto control plugin adjusts the exposure. This is clearly not the same histogram that I get by having the histogram calculated in eg. XnView or another program. Another thing peculiar about the histogram is that it looks like the blacks are crushed, even though it doesn’t look like that’s the case looking at the image (or the XnView histogram). I suspect I’m doing something wrong with how I set up the out parameters for the histogram - I’d be happy to learn what it is.
To answer your question on how to reproduce the issue: insert the above code into the auto control sample, and inspect the contents of histData[1] as the amount of light is changed in the room (I did this in a camera lab with dimable flicker-free LED-s)! The sensor is a Sony IMX 390.
To illustrate the issue, I’m posting these 2 screenshots, which have XnView’s histogram overlayed in the top left corner. In the lower left part of the screen is histogram[1] from the ISP (the RGB values are aggregated into luminance with a simple weighted average). The amount of light from the LED-s is 20% and 15% respectively.
