Turn off all corrections done by the ISP

[mlane]

I asked two questions previously but got no answers. So I’m back again asking for help. I have a TX2, Jetpack 3.1, IMX377, and an M12 lens. Here is my gstreamer pipeline (as python code):

gstreamer_str = ("nvcamerasrc wbmode=1 auto-exposure=1 edge-enhancement=0 tnr-strength=1 tnr-mode=2 flicker=0 intent=2 fpsRange='1.0 30.0' exposure-time=0.03 dump-bayer=true ! "
"video/x-raw(memory:NVMM), width=(int){sw}, height=(int){sh}, format=(string)NV12, framerate=(fraction)15/1 ! "
"nvvidconv ! video/x-raw, width=(int){ih}, height=(int){ih}, format=(string)GRAY8 ! "
"videoconvert ! appsink ").format(sw=sensor_width, sh=height, iw=image_width, ih=image_height)

Note that I’ve looked at the nvcamerasrc options and I’ve turned off or minimized the strength of everything that I can. Here are some example images. These aren’t the entire image, they’re crops of the same region of a target. The box is 50 micrometers across:

First the image that gets returned from the camera into opencv:

External Media

Here I’ve taken the raw bayer image and used opencv’s cvtColor function:

External Media

I would love to be able to grab the images straight from the ISP. This will be much faster and easier than writing the image out to disk as a raw file and then opening the file in opencv, but the images from the ISP are basically useless for us because of whatever corrections are going on behind the scenes. Incidentally, we’ve used the argus_camera application and turned off all the options and we get something even worse.

How can we minimize or completely turn off all of the “corrections” that the ISP is doing so that we can get images that we can actually use for our work? This is our main blocker at this point.

Thanks

[JerryChang]

hello mlane,

please refer to [Sensor Driver Programming Guide] in the release documentation.
there are two types of camera programming paths,

• Camera Core Library Interface
• Direct V4L2 Interface

if you’re going to bypass ISP, please choose V4L2 interface directly.
there’s v4l2-ctl for this approach.
for example,

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10 --stream-to=test.raw

[mlane]

Thanks for the response, but it doesn’t actually answer my question which was this:

How can we minimize or completely turn off all of the “corrections” that the ISP is doing so that we can get images that we can actually use for our work?

I’m able to get images from the ISP (currently we are using the gstreamer pipeline with nvcamerasrc from above, but we’d use libargus in production) and I’m able to bypass the ISP using the following v4l2-ctl command:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=4104,height=3046,pixelformat=RG12, --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test.raw

The issue is not so much can we get images, it is can we get useable images. What we’re getting from the ISP, even with all settings known to us turned off, is unusable garbage for our purposes. I’d love to be able to change that so we could use the ISP for debayering and maybe some other corrections instead of sending those to the CPU, but at the moment we can’t.

[JerryChang]

hello mlane,

we’ve used the argus_camera application and turned off all the options and we get something even worse.
may i know what’s the criteria about your use-case.
also what’s your expected result?

there are lots of works doing with ISP,
you may use v4l2-ctl to get raw files and perform de-mosaic by your own.
thanks

[mlane]

Hi JerryChang,

Our use case has to do with microscopic, or nearly microscopic imaging. We need fine detail on things that are very small and we cannot tolerate much in the way of noise or artificial sharpening or edge enhancements. Please compare the two images I posted in my original post to see what these kinds of corrections do to our images.

This image was captured with v4l2-ctl and de-mosaic’d in opencv:

External Media

This image is the output of the ISP using the command posted in my first comment above:

External Media

A further requirement is that we will likely be in a situation of needing the full 30fps throughput for 3, possibly more, CSI cameras.

Our expected result are images that the ISP has converted to grayscale without all the noise and miscellaneous artifacts (again, see the images above to get a sense of what I’m talking about).

there are lots of works doing with ISP,

Yes, indeed. We were very disappointed to find that the ISP is doing a lot to the image even when we are using the following settings in nvcamerasrc:

wbmode=1

auto-exposure=1

edge-enhancement=0

tnr-strength=1

tnr-mode=2

flicker=0

I’d like the ISP to do nothing at all if we choose and then to allow us to have fine-grained control over what the ISP is doing. We, for instance, would not want the ISP to try to de-mosaic an image that comes from a black and white sensor. And perhaps some level of edge enhancement, sharpening, noise reduction, etc. is tolerable, but we’d like to be able to adjust those to suit our needs.

you may use v4l2-ctl to get raw files and perform de-mosaic by your own.

We have no interest in replicating the work of the ISP if we don’t have to. We also are not interested in hacking our way around the limitation of being required to stream the raw file to disk since this will have significant performance consequences.

So is there any way of getting fine-grained control over what the ISP does? Is there any way we can speak to an engineer about this?

Thanks,
Mike Lane

[JerryChang]

hello mlane,

if you don’t have capability to handle raw files,
please contact our camera partners to request ISP tuning support, [url]https://developer.nvidia.com/embedded/community/partners[/url]
thanks

[mtanguay]

We are working with both Leopard and E-con on this issue and so far neither has been able to provide a solution.

[mtanguay]

Hi Jerry Chang,

We have exhausted our work with Leopard Imaging on this issue and they confirmed that they cannot bypass (turn off all features) on the ISP.

To further explain, we want to use the ISP but we are finding that some features such as sharpening are applied to all images that pass through the ISP. We do not want this.

Also, we know what to do with raw images that bypass the ISP but we are hoping to use the ISP in many cases. If we bypass the ISP we will simply go a different route and no longer consider using the TX2.

[alexisguiter]

Hi @mtanguay. I’m curious, have you managed to solve your issue?

I’m using a colour IMX296 with Libargus and the ISP onboard the Xavier NX. I too found that the oversharpening makes the image near un-usable for my purposes. I was told companies like D3 engineering could tune the ISP, but not sure if that includes turning off oversharpening.

@JerryChang Perhaps there is a way to turn off the oversharpening in gstreamer/libargus now? For reference here’s the pipeline I use:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 timeout=30 ispdigitalgainrange='1 1' gainrange='1 1'aeantibanding=0 wbmode=0 ee-mode=0 tnr-mode=0 ! 'video/x-raw(memory:NVMM),width=1440,height=1080,framerate=60/1,format=NV12' ! omxh265enc control-rate=0 qp-range=0,1:0,1:0,1 quant-i-frames=0 quant-p-frames=0 temporal-tradeoff=0 preset_level=3 ! matroskamux ! filesink location=test.mkv -e

[JerryChang]

hello alexisguiter,

this is a three years old topic, you may also note that the plugin is different.

may I know what’s the capture results of over-sharpening?
please refer to NVIDIA V4L2 Camera Description and Extensions, you’re setting ee-mode=0, which turn-off the edge enhancement;
this may be the reason why you’ll see turn off the oversharpening results.
thanks

[alexisguiter]

Hi!
I’ve attached 2 examples below, one filmed in the day, the other at dusk. You can see how grainy the footage is.

[JerryChang]

hello alexisguiter,

these grainy pixels should be able to remove by the noise reduction function,
however, it’s one of ISP feature, could you please initial a new forum discussion thread for the follow-up,
thanks

[alexisguiter]

Dear @JerryChang

The noise reduction is different. Oversharpening creates light halos around objects, this is not noise per se. Attached is a close up of a oversharpened feature. Notice the white halo around the wooden poll.

[JerryChang]

hello alexisguiter,

I see, please initial a new forum discussion thread,
we’ll follow-up the issue. thanks