DTS settings and image color saturation problem with multiple CSI cameras

There is a thread on TX1 board regarding image color saturation problem for a multiple cameras system. The issue seems not quite resolved yet. https://devtalk.nvidia.com/default/topic/1008695/isp-settings-and-ncam_cache-/
Recently, I experienced a similar issue on TX2.

The problem is that on the CSI-2 bus, multiple of same type cameras produce different result in color saturation, as shown by cospan’s image samples:

To my understanding, there is a set of cache for the cameras under /var/nvidia/nvcam/settings/

nvcam_cache0.bin
nvcam_cache1.bin
nvcam_cache2.bin
nvcam_cache3.bin
nvcam_cache4.bin
nvcam_cache5.bin

each corresponds to the settings of a camera on the CSI bus.

As cospan described, somehow nvcamera-daemon built bad caches for camera settings except for the first one.

The contents are very much different between the caches in binary comparison.

Interestingly, if you copy the nvcam_cache0.bin and use it to override the other cache files, you got a good result. The colors appear balanced. Apparently, this is not the approach supposedly to handle multiple cameras.

I checked the device tree, all the camera modules are defined in DTSI, and each has a unique badge name.

What is the requirement in DTSI or other settings in order to generate the cache properly?

@tmx3
Could you remove them to let it regen again to check the colors issue.

Hi ShaneCCC, thanks for the direction.
Our team did the test and here are the result:

Saved a copy of the "good" nvcam_cache_0.bin
The configuration is for six cameras, keep it is.
Delete all cache*.bin file under /var/nvidia/nvcam/settings/
Run terminal command (randomly pick one of the cameras): gst-launch-1.0 nvcamerasrc sensor-id=5 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv flip-method=2 ! nvegltransform ! nveglglessink -e
The execution re-creates all cache*.bin files automatically
Compare the files, all are same
Color is off (as if saturated)
Take the saved nvcam_cache_0.bin to replace all the caches.
Re-run the video, the color is good.

We couldn’t tell how the “good” nvcam_cache_0.bin was initially generated. It could be created and stayed since we started the test in configuration of a single camera, or it could be from the original default camera. That actually leads to a question regarding how is the cache file generated. We had to flash the device tree when change the camera set configuration. Here are two questions:

How is the cache file get generated?
Is it dependent on the sensor used?

Thanks in advance and happy holidays!

@tmx3
Is it possible the “good” nvcam_cache_0.bin generate from the reference sensor board ov5693.
Would you connect the ov5693 to regen to check it.

Hi ShaneCCC, we did some extensive tests and analysis, here are what we found:

Rebuilt the JetPack 3.1 on TX2, practically wipe out everything. The cache file /var/nvidia/nvcam/settings/nvcam_cache_0.bin was generated by the very first GST command and the cache file stays.
Found that re-building the kernel with new camera modules won't move the cache file.
Found that flashing new DTB from the Host, followed by GST command won't move the cache file.
If the cache file was removed manually, GST command will re-generate the cache.

We also noticed that

Need to remove the cache each time when switching DTB for different cameras
The keyword in the badge definition, such as "P5V27C" for build-in sensor plays a critical role in camera operation

Now I need you guys help in DTS and DTSi badge setting.

(a) Which keyword should be used for IMX219’s badge? As there are A815P2 in ref (1), P5V27C in ref (2), etc; any guideline for the selection?

(b) If I use keyword A815P2 in the badge for IMX219, the camera would adjust the exposure slowly and take a few seconds to settle on a rather dark display. So the question is how the keyword relates to the underline nvcamera-daemon/ISP stacks, and if anything else should be matched in order to fit the ISP and nvcam operation?

© Are those keywords created by certain coding method?

(d) Is there a naming convention for the badge in DTSi?

Reference:
(1) https://devtalk.nvidia.com/default/topic/1008695/isp-settings-and-ncam_cache-/
(2) https://devtalk.nvidia.com/default/topic/978143/jetson-tx1/whats-the-root-cause-of-new-sensor-not-working-with-nvcamerasrc/1
(3) https://devtalk.nvidia.com/default/topic/1010558/v4l2-drivers-for-tegra-x2-v4l2-and-nvcamerasrc/
(4) https://devtalk.nvidia.com/default/topic/1009359/jetson-tx1-sensor-driver-porting-to-tx2-4-4-kernel-/

Thanks in advance.

@tmx3
The badge info for the reference IMX219 sensor is A815P2, however it’s may not fixed to your sensor because the lens may not the same. If it’s different lens you need go through tuning process for this sensor. You need scaling partner’s help for the tuning process if want it.

Hi ShaneCCC, thanks for the information.

From TX2’s ISP/nvcam perspective, I’m not sure if I follow the idea of altering the badge keys to a code other than A815P2 when lens changes, say from standard 35mm to fisheye, etc… A requirement of using different key would lead to the existence of other parameters in the DTS settings in order to balance the result.

My tests also indicate that the badge setting is tightly coupled to the image processing and camera control.

As mentioned earlier, for example using IMX219 sensor, if I set the key to A815P2, the video takes about 5 seconds to gradually turn the picture brightness from over exposure to nightly dark. If I replace the key with an arbitrary string of characters , the exposure appears to be adjusted very quickly, but the color balance is slightly off. It appears as if a badge settings with a wrong keyword would simply disable the control.

Thanks again for the help.

@tmx3
The ISP configure for A815P2 IMX219 is totally different with your module. Our module is not fish eye lens. You should new a badge for your module and go through the tuning process to get great image quality.

Hi ShaneCCC, yes, I understand the point that different camera module needs a different configuration.

NVidia has published massive amount of data to promote the TX2 based eco-system. Sometime it’s hard to find the specific piece of information. Thanks you for the information about the importance of badge keyword in DTSi.

That leads to my next step of learning about the DTSi settings for TX2 applications. Let me list related questions for reference, not necessarily have to be resolved all the once.

How is the badge keyword, such as A815P2, related to the operation of TX2 camera ISP?
What's the guideline about the device tree coding for Jetson CSI-2 cameras, particularly to the section of tegra-camera-platform in DTSi ?
What's the best way to force the GStreamer pipeline to refresh the nvcam cache, other than manually remove the caches under /var/nvidia/nvcam/settings?
  1. After tuning process the ISP configure file will gen and this configure file have this badge information for loading these config.
  2. You can check the sensor programing guide from l4t document.
  3. No better solution than manually now.

@ShaneCC, thanks for your tip. I’ll check the l4t document as guided.

@tmx3,
How did this end, did you find a way to fix it ?

I am having the same problem with two imx219 cameras.

One camera seems to be almost ok, a little tinted pink.

The other cam uses a long time to stabilize and ends up with a much worse picture than the first one…

This are raspberry pi V2.1 imx219 cameras on TX2 JetPack 28.1

@tmx3 @ShaneCC @Spawn32

I’m also having a similar issue with 4 CSI cameras where some feeds have a pink tint to them. If I change the badge it doesn’t seem to affect much.

How was this resolved?

Any update on this? I’m using A815P2, and my video starts normal and exposure gradually saturates. If I use P5V27C, this happens:

As far as I know, I’m using a standard off the shelf Rpi V2.1 (IMX219) camera

We tried badge A815P2 and encountered similar issues like exposure. Still, A815P2 seems the best for IMX219 so far. Would like to hear if any better badge code for the sensor.

Frankly, given the raw camera modules on hand with all kind of manufacturing issues and driver settings, the color tuning becomes a darkroom casting on camera makers in finding the resolution. It makes us rather difficult to determine whether the problems were caused by manufacturing defects, driver bugs, register initialization, or something had to be handled by the down stream development such as color tuning.

What I’d hope is that nvidia could open some sort of commands to the developers for a basic level of color tuning. Once the hardware had been confirmed functioning properly, we would be in a better position to acquire the funding and resources to work with the scaling partners for further enhancement.

Apart from the exposure problems, are you able to obtain a crisp image? Mine seems very unfocused and blurry.

@rm95, the sharpness seems normal. If you are using the Pi Cam like we do, you may adjust the focus of IMX219 with the tool (it comes with Raspberry Pi RS version module for free).
Image of the lens adjustment tool (by Adafruit) https://www.adafruit.com/product/3518?gclid=EAIaIQobChMIoZ6dvJmk4AIVyLbACh0TAw_GEAYYASABEgLA6vD_BwE

Okay, I’ll check if I have it, thank you.

Btw, what driver files are you using? I found these by Cospan: http://cospandesign.github.io/assets/files/posts/rpi-camera/imx219.c
http://cospandesign.github.io/assets/files/posts/rpi-camera/imx219_mode_tbls.h

Are these the same you are using as well?

We had to tweak the c file and header according to IMX219 spec, pretty much try-and-error. For example, we found that the setting of initialization for
/* Coarse Integration Time /
{0x015A, 0x04},
{0x015B, 0x22},
works better that the original
/
Coarse Time */
{0x015A, 0x0D}, {0x015B, 0x05},

We also added a mode for 3280x2464 @21 fps. It’s an on-going process.

Could you share the driver files? I will DM you my email ID if you can.

Also, what L4T version are you running it on?