Turn off all corrections done by the ISP

I asked two questions previously but got no answers. So I’m back again asking for help. I have a TX2, Jetpack 3.1, IMX377, and an M12 lens. Here is my gstreamer pipeline (as python code):

gstreamer_str = ("nvcamerasrc wbmode=1 auto-exposure=1 edge-enhancement=0 tnr-strength=1 tnr-mode=2 flicker=0 intent=2 fpsRange='1.0 30.0' exposure-time=0.03 dump-bayer=true ! "
"video/x-raw(memory:NVMM), width=(int){sw}, height=(int){sh}, format=(string)NV12, framerate=(fraction)15/1 ! "
"nvvidconv ! video/x-raw, width=(int){ih}, height=(int){ih}, format=(string)GRAY8 ! "
"videoconvert ! appsink ").format(sw=sensor_width, sh=height, iw=image_width, ih=image_height)

Note that I’ve looked at the nvcamerasrc options and I’ve turned off or minimized the strength of everything that I can. Here are some example images. These aren’t the entire image, they’re crops of the same region of a target. The box is 50 micrometers across:

First the image that gets returned from the camera into opencv:

Here I’ve taken the raw bayer image and used opencv’s cvtColor function:

I would love to be able to grab the images straight from the ISP. This will be much faster and easier than writing the image out to disk as a raw file and then opening the file in opencv, but the images from the ISP are basically useless for us because of whatever corrections are going on behind the scenes. Incidentally, we’ve used the argus_camera application and turned off all the options and we get something even worse.

How can we minimize or completely turn off all of the “corrections” that the ISP is doing so that we can get images that we can actually use for our work? This is our main blocker at this point.

Thanks

hello mlane,

please refer to [Sensor Driver Programming Guide] in the release documentation.
there are two types of camera programming paths,

  • Camera Core Library Interface
  • Direct V4L2 Interface

if you’re going to bypass ISP, please choose V4L2 interface directly.
there’s v4l2-ctl for this approach.
for example,

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10 --stream-to=test.raw

Thanks for the response, but it doesn’t actually answer my question which was this:

How can we minimize or completely turn off all of the “corrections” that the ISP is doing so that we can get images that we can actually use for our work?

I’m able to get images from the ISP (currently we are using the gstreamer pipeline with nvcamerasrc from above, but we’d use libargus in production) and I’m able to bypass the ISP using the following v4l2-ctl command:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=4104,height=3046,pixelformat=RG12, --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test.raw

The issue is not so much can we get images, it is can we get useable images. What we’re getting from the ISP, even with all settings known to us turned off, is unusable garbage for our purposes. I’d love to be able to change that so we could use the ISP for debayering and maybe some other corrections instead of sending those to the CPU, but at the moment we can’t.

hello mlane,

we’ve used the argus_camera application and turned off all the options and we get something even worse.
may i know what’s the criteria about your use-case.
also what’s your expected result?

there are lots of works doing with ISP,
you may use v4l2-ctl to get raw files and perform de-mosaic by your own.
thanks

Hi JerryChang,

Our use case has to do with microscopic, or nearly microscopic imaging. We need fine detail on things that are very small and we cannot tolerate much in the way of noise or artificial sharpening or edge enhancements. Please compare the two images I posted in my original post to see what these kinds of corrections do to our images.

This image was captured with v4l2-ctl and de-mosaic’d in opencv:

This image is the output of the ISP using the command posted in my first comment above:

A further requirement is that we will likely be in a situation of needing the full 30fps throughput for 3, possibly more, CSI cameras.

Our expected result are images that the ISP has converted to grayscale without all the noise and miscellaneous artifacts (again, see the images above to get a sense of what I’m talking about).

there are lots of works doing with ISP,

Yes, indeed. We were very disappointed to find that the ISP is doing a lot to the image even when we are using the following settings in nvcamerasrc:

  • wbmode=1
  • auto-exposure=1
  • edge-enhancement=0
  • tnr-strength=1
  • tnr-mode=2
  • flicker=0
  • I’d like the ISP to do nothing at all if we choose and then to allow us to have fine-grained control over what the ISP is doing. We, for instance, would not want the ISP to try to de-mosaic an image that comes from a black and white sensor. And perhaps some level of edge enhancement, sharpening, noise reduction, etc. is tolerable, but we’d like to be able to adjust those to suit our needs.

    you may use v4l2-ctl to get raw files and perform de-mosaic by your own.

    We have no interest in replicating the work of the ISP if we don’t have to. We also are not interested in hacking our way around the limitation of being required to stream the raw file to disk since this will have significant performance consequences.

    So is there any way of getting fine-grained control over what the ISP does? Is there any way we can speak to an engineer about this?

    Thanks,
    Mike Lane

    hello mlane,

    if you don’t have capability to handle raw files,
    please contact our camera partners to request ISP tuning support, https://developer.nvidia.com/embedded/community/partners
    thanks

    We are working with both Leopard and E-con on this issue and so far neither has been able to provide a solution.

    Hi Jerry Chang,

    We have exhausted our work with Leopard Imaging on this issue and they confirmed that they cannot bypass (turn off all features) on the ISP.

    To further explain, we want to use the ISP but we are finding that some features such as sharpening are applied to all images that pass through the ISP. We do not want this.

    Also, we know what to do with raw images that bypass the ISP but we are hoping to use the ISP in many cases. If we bypass the ISP we will simply go a different route and no longer consider using the TX2.