Can I turn off or reduce Argus ISP image sharpening (edge enhancement)

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
,
1

I know this topic has been discussed before (e.g., here and here), but a few years have passed and I am still running into similar issues.

Here’s a brief description of my set-up:

  • Xavier NX on a CTI Boson carrier
  • Framos FSM-IMX283 CSI module + 2 x FSM-IMX264 CSI modules
  • Running CTI Boson BSP based on Jetson Linux R32.7.3 (Jetpack 4.6.3)
  • I have installed the basic Framos .isp tuning profiles, but I disabled lens shading correction
  • I have a custom application using libArgus to capture a stream in either PIXEL_FMT_YCbCr_444_888, PIXEL_FMT_YCbCr_420_888 or PIXEL_FMT_RAW16 formats
  • I have the stream resolution set to the full resolution of the sensor (according to device tree and sensor datasheet), e.g., 5496 x 3694 for the IMX283.
  • In addition to the .isp tuning profiles I have modified the edge enhancement settings, EDGE_ENHANCE_MODE_OFF or EDGE_ENHANCE_MODE_HIGH_QUALITY with various strength settings, including strength 0 (and event negative values, for completeness!)
  • I have also tried removing the .isp tuning profiles (and deleting the cached .bin files); I can see from the nvargus-daemon logs if the profiles are actually being used or not.

Despite all this I cannot reduce the amount of edge enhancement below a certain base level, so either there is an additional setting that is perhaps not exposed via the libArgus API, or there is a large amount of sharpening baked into the ISP Bayer demosaicing algorithm, or perhaps I end up with some inadvertent resizing of the image leading to resampling with a filter that includes a contrast boost.

Here’s a slanted-edge measurement showing the spatial frequency response (SFR) of an image captured as YUV444 using the Argus ISP (blue SFRcurve), vs the same scene captured as RAW16 and then demosaiced (software DFPD method) shown as the green curve, and lastly the RAW16 software image with a small amount of post-capture sharpening (orange curve).

slanted_edge_sfr
slanted_edge_sfr1807×661 287 KB

The YUV444 Argus ISP image has an excessive amount of sharpening, with SFR contrast well above 1.4. This also shows up as visible undershoot and overshoot if we look at the image intensity profile across the edge (the edge spread function):

slanted_edge_esf
slanted_edge_esf855×579 126 KB

Again, note the YUV444 Argus ISP image is the blue curve, with the green and orange curves corresponding to a RAW16 capture with software processing (demosaicing, WB, CCM, etc.).

Here’s a magnified view of the YUV444 Argus ISP image (4x nearest-neighbour interpolation), where the ringing is clearly visible:

argus_isp_crop
argus_isp_crop736×520 101 KB

compared to the RAW16 software processed image:
software_crop
software_crop736×520 83.3 KB

Given that I am using a much more recent JetPack version, I would assume that previous issues with libnvscf have been resolved. So back to my topic questions: Is it possible to further reduce the amount of edge enhancement when using YUV444 (or YUV420) capture through the Argus ISP, beyond what I see with EDGE_ENHANCE_MODE_OFF ?

Or is there a trick to capturing a full-sensor-resolution stream to avoid any resizing / resampling in the Argus ISP, other than setting the stream resolution to the sensor resolution? (this seems to work as expected with RAW16, anyway)

3

Try add below to the camera_overrides.isp at /var/nvidia/nvcam/settings/
If the file doesn’t present create it and change mode to 644 by chmod 644 camera_overrides.isp

sharpness.v5.enable = FALSE;
4

Thanks Shane!

I have tried the sharpness.v5.enable = FALSE; setting, and I can see that switching it to TRUE increases the amount of sharpening.

However, even setting it to FALSE still results in images that have visible ringing (undershoot / overshoot) near edges.

Looking at the nvargus-daemon logs I noticed there were some entries tagged as “Ltm”, which appears to translate to Local Tone Mapping, i.e., similar to a Clarity post-processing step. If there was indeed a Local Tone Mapping post-processing step enabled it would explain the visible ringing.

I scanned libnvscf.so for strings and found keywords like localtonemap.v5.ModifyEnable, so I tried the following settings:

localtonemap.v5.DynamicRange = 10;
localtonemap.v5.LumaBlendEnabled = FALSE;
localtonemap.v5.ModifyEnable = 0;

This produced nvargus-daemon logs containing

Ltm ISP5 settins: ModifyEnabled 1
Ltm ISP5 settins: DynamicRange 10

so at least the DynamicRange setting is taking effect, but the ModifyEnable(d) setting appears to remain set to “1” in the logs.

Anyway, none of this appears to have reduced the visible ringing (still clearly visible in provided apps like yuvOneShot’s output, so it’s not my application). But this does suggest that there are potentially other post-processing steps, e.g.,

  • Local Tone Mapping
  • Clarity
  • Contrast Enhancement

that appear as keywords in libnvscf.so that could be responsible for the ringing. Could you please provide some suggested settings for the .isp file to disable these post-processing steps in addition to the sharpening?

It would be great if nvidia could provide a “null” .isp setting that zeroes out all post-processing for testing.

5

What’s the BSP version?

cat /etc/nv_tegra_release

6

# R32 (release), REVISION: 7.3, GCID: 31982016, BOARD: t186ref, EABI: aarch64, DATE: Tue Nov 22 17:32:54 UTC 2022

7

Did you apply any camera_overrides.isp file? /var/nvidia/nvcam/settings/camera_overrides.isp

Could you attach here to check.

Thanks

8

Sure!

I have some black level and ccm settings added in there to improve the output image color (when using yuvOneshot, for example), but the excessive local contrast enhancement is still visible without those settings.

I also left in my attempt to disable the local tone mapping, but those settings do not appear to have any effect anyway.

camera_overrides.isp.txt (694 Bytes)

9

Hi Shane!

I posted my camera_overrides.isp file above. Do you perhaps have any updates for me with regards to reducing / disabling Local Tone Mapping (or related methods) in the Argus ISP pipeline?

Thanks!

10

Could you migrate to r35.4.1 due to the LTM have much update for this release.

Thanks

11

Ok, this will be a little hard with my main set-up (CTI Boson with Framos modules) because CTI has not yet released a Boson BSP for JetPack 5.1.2.

But I also have a Xavier NX dev-kit, so I’ll source something like a Raspberry Pi v2 camera to do an A/B test between JetPack 4.x and 5.1.2 on the dev-kit.

Hopefully I will be able to get back to you by next week with an update.

12

Hi Shane,

I have obtained some new data using Raspberry Pi Camera v2 (IMX219) and a regular nvidia Xavier NX dev-kit, using the recommended JetPack 5.1.2, i.e.,

cat /etc/nv_tegra_release
# R35 (release), REVISION: 4.1, GCID: 33958178, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug  1 19:57:35 UTC 2023

using minimal overrides

cat /var/nvidia/nvcam/settings/camera_overrides.isp
sharpness.v5.enable = FALSE;

and the Argus oneShot sample program, modified only to lock down the exposure (fixed exposure time, gain, and isp digitial gain) and to disable noise reduction (DENOISE_MODE_OFF) to minimize any non-linear processing. Exposure was verified using a ColorChecker mid-gray patch yielding 8-bit RGB values of roughly 120. Ignoring the white balance, I still see the same kind of local contrast enhancement artifacts: a black rectangle inside the target, and a white rectangle around it, neither of which are actually present in the physical scene:

imx219_argus_isp
imx219_argus_isp612×560 86.5 KB

I have tried fiddling with all the localtonemap settings in the .isp file, but nothing seems to reduce the artifacts.

Would it be possible to ask the Argus ISP team if they can identify the processing step that produces these artifacts? I really would appreciate it greatly if there is a way to eliminate the artifacts.

14

@frans1
I was reported to IQ team to check it.

Thanks

15

Hi Shane, just following up to hear if the IQ team have provided you with any feedback yet.

Thanks!

16

This question has been raised by many people. There can be two reasons. The first is the issue of ISP adjustment. The second one is caused by the incorrect matching of the camera module lens and sensor.

The simplest way to distinguish is to determine whether this phenomenon still occurs after turning off the ISP.
Of course, you can also observe whether RAW also has this problem.

17

Hi James,

I have reproduced the artifacts on three different camera modules: Framos FSM-IMX264, Framos FSM-IMX283 and Raspberry Pi v2 camera module (IMX219). When I let the Argus ISP process the image, then the overshoot/undershoot is clearly visible, but when I process the image myself from RAW16 there is no overshoot/undershoot (see example in my original post at the top of this thread).

I do think that we have made some progress here, since the artifacts are not caused by the sharpening parameters (edge enhancement in libArgus API); by turning on the edge enhancement feature I can make it worse …

So the artifacts appear to be caused by some other processing step that has no obvious means of adjustment through the libArgus API. I suspect that the culprit is most likely Local Tone Mapping, but I am hoping that someone from the Argus ISP IQ team will provide some more insight.

18

Colud you share RAW ?

19

Sure, here is a crop of the same general region as shown in the first post above. The crop is Bayer-aligned, so it is still in RGGB ordering, and please note that the values have been shifted right already to fit in the 12-bit range 0-4095.
crop_raw

20

Hi Shane,

Do perhaps have any feedback on this topic from the IQ team?

Thanking you
-Frans

21

Hi Shane,

It has been a while, so I was hoping that you might have an update from the IQ team on this matter.

I did notice that this issue has been tagged “nvbugs” in December 2023, so I’m just wondering if this means the issue is scheduled to be fixed?

Thanking you

Regards,
-Frans

22

Please replace the attached lib to try.

libnvscf.so.sharpness (8.4 MB)