Ringing (edge enhancement) when stream configured to YCbCr420_8888

BareMetalCoder · December 19, 2019, 9:20pm

We are seeing ringing (like edge enhancement, an artificial exaggeration/white halo around sharp white/black transitions) in our images when we set our stream to YCbCr420 with the Argus examples on the Jetson Xavier. The camera is an IMX577 module from Leopard Imaging and a fisheye lens.

The Jetpack version is 4.2.0, which is the version we are stuck with for the moment, given the sensor/MIPI card combo.

Pronounced black/white transitions in the image appear to be submitted to strong edge enhancement set to a very aggressive setting. If we grab the images as RAW16, and debayer ourselves using CUDA, we don’t see the ringing, which leads me to believe that something between the CameraDevice, CameraSession and OutputStream is introducing this effect.

In the ringing case, we instruct the stream to use the YCbCr420 Pixel Format:

iEGLStreamSettings->setPixelFormat(Argus::PIXEL_FMT_YCbCr_420_888);

We also try to disable everything we can that might pertain to edge enhancement:

iSourceSettings->setFrameDurationRange(Argus::Range<uint64_t>(frameDuration));
iSourceSettings->setExposureTimeRange(1e6*20);
iSourceSettings->setGainRange(1.0f);
iAutoControlSettings->setExposureCompensation(0.0f);
iAutoControlSettings->setIspDigitalGainRange(1.0f);
iDenoiseSettings->setDenoiseMode(Argus::DENOISE_MODE_FAST);
iDenoiseSettings->setDenoiseStrength(0.4f);
iEdgeEnhanceSettings->setEdgeEnhanceMode(Argus::EDGE_ENHANCE_MODE_OFF);
iEdgeEnhanceSettings->setEdgeEnhanceStrength(0.0f);

Disabling post processing has no effect:

iStreamSettings->setPostProcessingEnable(false);

I have tried the argus_camera application, and I haven’t been able to shut this effect off with any of the settings. Incidentally, changing the edge enhancement settings don’t seem to have an effect at all.

Does anyone know of any other settings we might need to turn off? I have gone through the list of interfaces in the documentation and I have run out of ideas – I must be missing something obvious…

BareMetalCoder · December 19, 2019, 9:35pm

Could it be due to something in the .isp file?

I’m looking at the sharpness settings here, not sure what they do. They all look like:

sharpness.v2.Preview[0] = {3, 3, 5, 5, 5, 5, 5}
...
sharpness.v2.Still[0] = {3, 3, 5, 5, 5, 5, 5}
...
sharpness.v2.Video[0] = {3, 3, 5, 5, 5, 5, 5} 
...

ShaneCCC · December 20, 2019, 3:37am

Does the problem only show when configure to YCbCr_420_888?

BareMetalCoder · December 20, 2019, 5:19am

I have only tried YCbCr420_888 and RAW16. When I use RAW16 and I debayer using CUDA, I don’t see the ringing.

ShaneCCC · December 20, 2019, 5:40am

You may need to attached the picture to tell the problem.

BareMetalCoder · December 20, 2019, 4:09pm

Here are some pictures attached. They are comparisons of the same features, captured using two different modes:

Left: YCbCr420_888, with the ringing
Right: RAW16, debayered in CUDA (no white balance adjustment, so it is normal for the colors to be a bit off). No ringing

External Media External Media

SimonZhu · December 23, 2019, 1:20am

Hi BareMetalCoder,
We can get good images via argus or gstreamer. Please give it a try.

argus_camera --device=0

And

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, framerate=29/1'  ! nvvidconv flip-method=0 ! 'video/x-raw, format=(string)I420' ! xvimagesink -e

BareMetalCoder · December 23, 2019, 6:45am

argus_camera shows ringing. We will look into gst-launch on Monday.

Also possibly worth mentioning: We are using a fisheye lens and the entire fisheye circle is contained in the image. Thus, there are about 500+ columns of “black” on both the left and the right of the image. Not sure if this could somehow throw things off on the Argus side.

BareMetalCoder · December 24, 2019, 4:56am

@ShaneCCC: Any insight? I don’t really suspect the camera because I can get non-ringing images if I capture raw and debayer manually.

Any way for us to know what is going on in the VI, even if it is basic information? It feels like a bug because changing the edge enhancement settings seems to have no effect. It almost feels like it is stuck at a very aggressive setting.

ShaneCCC · December 24, 2019, 5:03am

It’s doesn’t matter with VI, it could be the ISP setting/tuning parameter cause the problem.
Could you have argsu_camera to disable the noise reduction/edge enhancement to confirm.

Also what’s the sensor output resolution and display resolution?

BareMetalCoder · December 25, 2019, 5:22pm

Thanks for the response. Sorry, I meant ISP, not VI.

Changing the noise reduction does not affect the ringing significantly. The images taken above are at Noise Reduction = 0.4, fast mode.

As for the edge enhancement, as I mentioned in the first post: it is turned off. That being said, turning it off or on doesn’t seem to produce any kind of visual difference. Changing the level when it is on doesn’t seem to have an effect either, which makes it seem like it is somehow “stuck”. That is what prompted me to reach out on this forum.

To reiterate: this ringing is not visible when I switch the pixel format to RAW16 instead of YCbCr420_888.

Were there any known issues with this feature or the ISP in general in Jetpack 4.2.0? It’s an older version, but the camera driver we have requires us to use this version for the moment. We are working on getting a system together to try Jetpack 4.2.2 or later.

Are there any automatic processes that determine the minimum / maximum level of enhancement in the ISP? Could they be influenced by the fact that we have a fisheye image that doesn’t completely cover the sensor?

Thanks

BareMetalCoder · January 6, 2020, 5:13pm

@ShaneCCC, Any news with this? I have provided all of the information you requested and this issue is critical for our applications. Thanks.

ShaneCCC · January 7, 2020, 3:00am

Could you remove all of the files in the /var/nvidia/nvcam/settings/ to try.

BareMetalCoder · January 7, 2020, 4:59pm

The image colors change somewhat because the tuning parameters are no longer there, but the ringing is the same as before.

Using YcbCr420_888 -> Ringing is still there
Using RAW16, Debayering manually in CUDA -> No ringing

I tried it in argus_camera (default settings) and in my own code at 1:1 (no scale-down) to see the exact pixels.

As before, changing the edge enhancement settings seems to have no effect.
→ Is it possible that there is a bug with this feature? ←

Reminder:

Using Jetpack 4.2.0 (R32.1)
IMX577 sensor (pin compatible with IMX477, uses IMX477 driver)

ShaneCCC · January 8, 2020, 3:17am

@BareMetalCoder
It could be better to sync up this issue with Leopard.

BareMetalCoder · January 8, 2020, 5:06pm

@ShaneCCC, I already spent several days with Leopard before I started this thread. We have explored this issue in depth with Simon’s excellent service, and all clues point to the ISP.

I have pointed to several strange things in this thread so far, and I don’t yet have an explanation from NVIDIA regarding why they may be occurring.

a) The Edge Enhancement settings from our code or other utilities such as argus_camera seem to be ignored, and…
b) The Edge Enhancement settings seem to be turned way up, with no way to turn them down or off.

Until they are explained, I don’t see how Leopard could have anything to do with this issue.

As I mentioned previously,

Based on my inspection of Leopard's driver, my undestanding is that it provides a bayered image to the Argus pipeline
We have confirmed that the RAW16 (bayered) output from the Leopard camera DOES NOT have ringing
Thus, the ringing/edge enhancement must be introduced at some point in the ISP, unless you are saying that the edge enhancement (and ISP feature) is being implemented in the Leopard driver.
You have not yet provided a possible failure mode that would explain why the Edge Enhancement settings seem to be ignored. Do you believe Leopard could be responsible for this? If so, please let us know how this could be possible so that we can continue the conversation with them - how could their driver be responsible for this? We have already established that their camera_overrides.isp file is not a culprit.

We have experienced engineering staff on both sides of this issue and all clues point to the black box in the ISP. Please help us by providing a possible explanation for these discrepancies. This may require an escalation to ther colleagues at NVIDIA.

Thanks

BareMetalCoder · January 8, 2020, 5:46pm

I have also found this older issue that seems similar, but is not marked as resolved. Do you know if this was addressed in a version after Jetpack 4.2?

https://devtalk.nvidia.com/default/topic/1031510/jetson-tx2/turn-off-all-corrections-done-by-the-isp/post/5248358/

ShaneCCC · January 9, 2020, 3:18am

It’s better to have Leopard to repo your problem and have co-work with nvidia to get help.

SimonZhu · January 21, 2020, 10:56am

Hi BareMetalCoder,
With Nvidia’s help, we got below ISP file. Please give it a try.

BareMetalCoder · January 21, 2020, 6:04pm

Thanks to Leopard’s cooperation on this issue, this ISP file for the IMX577 significantly reduces the ringing and also improves the noise.

We will evaluate it and I will mark this as an answer if it meets our requirements.
Thanks.