ISP Settings and 'ncam_cache'

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Until recently I’ve been using an IMX219 camera sensor on the TX1 without issue. A couple of weeks ago I was attempting use gstreamer to capture video from different boards (circuit board with TX1 module) when one of the TX1 module with a camera sensor started acting strange.

On the misbehaving TX1 I could capture a raw bayer image from the camera sensor and use some userland applications to convert it to a png and the image looked fine but when I would use nvgstcapture or a gstreamer pipeline the image would come out oversaturated with one of the following: blue, red or yellow, it was like a grayscale image except instead of white as the brightest it would be blue.

I have two TX1 modules both with different rootfs, call them A and B for reference.

I can flash both TX1s with rootfs ‘A’ and they would both capture images correctly.
I can flash both TX1s with rootfs ‘B’ and they would both fail.

This proves it’s not a hardware issue (Circuit board or the TX1 Module).

If one TX1 had ‘A’ and the other had ‘B’ and I installed the same bootloader, kernel and DTB I would see the same result (‘A’ would work and ‘B’ would fail)!?

The one difference I noticed between ‘A’ and ‘B’ was if I ran ‘nvcamera-daemon’ as a foreground application and run the following gstreamer command on both:

gst-launch-1.0 -v -m nvcamerasrc sensor-id=0 fpsRange="30 30" num-buffers=1 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw, width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! jpegenc ! filesink location=/home/ubuntu/Projects/video-tester/test.jpg -e

I would see differing outputs as follows:

‘A’ (Working):

NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)

‘B’ (Not Working):

NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)

Note the: LSC:patch width less than 12

When I run a different pipeline to capture a video stream that above error would repeat continuously

I won’t go into the painful details about using a hex editor to view where those errors come from but I finally found something that would effect the behavior of the cameras.

in the /var/nvidia/nvcam/settings/ directory if I replaced the nvcam_cache*.bin from ‘B’ with the content of the same file within ‘A’ I wouldn’t see that error. The camera sensors would work fine and the problem is fixed.

If I delete the files and run the applications again the error would arise.

It looks like these files store the settings for the ISP for use with a camera. Unfortunately there is something going weird when the application needs to populate the camera’s cache:

Here is an excerpt from the nvcamera-daemon when I delete the cache files and re-run the jpeg capture pipeline.

...
ISP Config file did not load from cache, quering deviceNvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 170: Error: Invalid isp config attribute element: "flash.TorchLevel[0]=100.00"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 171: Error: Invalid isp config attribute element: "flash.FlashLevel[0]=180.00"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 180: Error: Invalid isp config attribute: "nr.v1.luma.Enable=TRUE"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 181: Error: Invalid isp config attribute: "nr.v1.luma.IsoThreshold=100"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 182: Error: Invalid isp config attribute: "nr.v1.luma.Slope=0.01"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 183: Error: Invalid isp config attribute: "nr.v1.luma.Threshold=0.015"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 184: Error: Invalid isp config attribute: "nr.v1.chroma.Enable=TRUE"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 185: Error: Invalid isp config attribute: "nr.v1.chroma.IsoThreshold=100"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 186: Error: Invalid isp config attribute: "nr.v1.chroma.Slope=0.40"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 187: Error: Invalid isp config attribute: "nr.v1.chroma.Threshold=0.08"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 188: Error: Invalid isp config attribute: "nr.v2.chroma.Enable=FALSE"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 189: Error: Invalid isp config attribute: "nr.v2.chroma.IsoThreshold=100"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 190: Error: Invalid isp config attribute: "nr.v2.chroma.Slope=1.0"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 191: Error: Invalid isp config attribute: "nr.v2.chroma.Threshold=0.0"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 192: Error: Invalid isp config attribute: "nr.v2.chroma.PredefinedGainResponse={0,0,0,0,0,0,1,2,3,4,4,4,4,4,4,4}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 193: Error: Invalid isp config attribute element: "ae.noiseReduction.Preview[0]={0,0,1,2,2,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 194: Error: Invalid isp config attribute element: "ae.noiseReduction.Preview[1]={0,1,1,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 195: Error: Invalid isp config attribute element: "ae.noiseReduction.Preview[2]={1,1,2,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 196: Error: Invalid isp config attribute element: "ae.noiseReduction.Preview[3]={2,2,2,3,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 197: Error: Invalid isp config attribute element: "ae.noiseReduction.Still[0]={0,0,1,2,2,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 198: Error: Invalid isp config attribute element: "ae.noiseReduction.Still[1]={0,1,1,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 199: Error: Invalid isp config attribute element: "ae.noiseReduction.Still[2]={1,1,2,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 200: Error: Invalid isp config attribute element: "ae.noiseReduction.Still[3]={2,2,2,3,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 201: Error: Invalid isp config attribute element: "ae.noiseReduction.Video[0]={0,0,1,2,2,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 202: Error: Invalid isp config attribute element: "ae.noiseReduction.Video[1]={0,1,1,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 203: Error: Invalid isp config attribute element: "ae.noiseReduction.Video[2]={1,1,2,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 204: Error: Invalid isp config attribute element: "ae.noiseReduction.Video[3]={2,2,2,3,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 205: Error: Invalid isp config attribute: "edgeEnhance.strength.Preview=0.113"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 206: Error: Invalid isp config attribute: "edgeEnhance.strength.Still=0.113"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 207: Error: Invalid isp config attribute: "edgeEnhance.strength.Video=0.113"
LSC:patch width less than 12
...

I’ve been searching around for a ‘sensor.cfg’ file but couldn’t find it.

I would think that the dts file for the camera would be the thing used to populate this cache file but it doesn’t seem as though the changes I make within the DTS file fixes this problem.

How can I configure the ISPs correctly whenever the cache needs to be rebuilt?

I don’t have access to the NVIDIA partners program so all of these files are opaque to me.

Thanks in advance for any help,

Dave

2

Hi cospan

Does this sensor module from Leaopard or some others vendor? I guess you can contact them to get the isp config file for you problem.

3

Shane,

Thanks for the feedback! No the driver is not from a vendor, I wrote it.

It would like to have access to adapt this sensor but barring that I would be happy if someone on the inside would let me know what values I have to set in order to get the nvcam_cach*.bin to build correctly.

If you would like a reference to the driver, mode table and dtsi file for the camera I’ve put them here.

Camera DTSI File: [url]https://raw.githubusercontent.com/CospanDesign/nvidia-tx1-kernel/mit-uav-revb/arch/arm64/boot/dts/tegra210-platforms/tegra210-camera-mit-uav-revb.dtsi[/url]
Driver: [url]https://raw.githubusercontent.com/CospanDesign/nvidia-tx1-kernel/mit-uav-revb/drivers/media/i2c/imx219.c[/url]
Mode Tables: [url]https://raw.githubusercontent.com/CospanDesign/nvidia-tx1-kernel/mit-uav-revb/drivers/media/i2c/imx219_mode_tbls.h[/url]

As a note I have three cameras attached to the board and I use a GPIO I2C multiplexer to access different cameras.

If possible perhaps someone could let me know what values on the dtsi file I could change in order to make this work.

4

Ｈi cospan
You can try to modify the badge info exactly the same like tegra210-camera-e3322-a00.dtsi to check if it can load the ISP setting for imx219. Otherwise you need scaling partner to help to tuning to gen an isp file.

module1 {
457				badge = "e3322_centerleft_A815P2";
458				position = "centerleft";
459				orientation = "1";
460				drivernode0 {
461					pcl_id = "v4l2_sensor";
462					devname = "imx219 31-0010";
463					proc-device-tree = "/proc/device-tree/host1x/i2c@546c0000/tca9548@77/i2c@1/imx219_b@10";
464				};
465			};
466			module2 {
467				badge = "e3322_centerleft_A815P2";
468				position = "centerright";
469				orientation = "1";
470				drivernode0 {
471					pcl_id = "v4l2_sensor";
472					devname = "imx219 32-0010";
473					proc-device-tree = "/proc/device-tree/host1x/i2c@546c0000/tca9548@77/i2c@2/imx219_c@10";
474				};
475			};
476			module3 {
477				badge = "e3322_centerleft_A815P2";
478				position = "topleft";
479				orientation = "1";
480				drivernode0 {
481					pcl_id = "v4l2_sensor";
482					devname = "imx219 33-0010";
483					proc-device-tree = "/proc/device-tree/host1x/i2c@546c0000/tca9548@77/i2c@3/imx219_d@10";
484				};
485			};
5

Shane,

Thanks for looking into this, that was a good idea, I should have thought of it.

I tried it out but no good, the nvcamera-daemon rebuilt the cache with bad values again. I grabbed the previous (working) nvcam_cache_0.bin and the one that was built and can see the difference but everything is in a binary format and it wouldn’t be trivial to reverse engineer it.

I ran an experiment:

I wanted to see how the ‘nvcam_cache_*.bin’ files were generated relative to the settings within the DTS:tegra_camera_platform entries.

Now I have the following three nvcam_cache0.bin files:

  • The old, working version
  • Version using my original values
  • Version with values from E3322 board values

I ran a diff of each of the files and came out with this result:

1 and 2 are different
1 and 3 are different
2 and 3 are the same?!

I would have thought that the nvcam_cache files generated from two different tegra-camera-platform groups would have yielded different cache files. There is something I’m not understanding.

From this I can conclude one of two things is happening:

  1. There is something wrong with the way I'm writing my DTS file
  2. The userland ISP application is independent of the 'tegra-camera-platform' entry within DTS file.

Regarding ‘1’:

I looked in other DTS files within the tegra210-platform directory in order to determine if there is something else that I should be putting inside there. I’ve looked inside ‘tegra210-platform/tegra210-jetson-cv-camera-modules.dtsi’ file and made sure that my ‘tegra-camera-platform’ looks exactly the same, even adding in the ‘tcp:tegra-camera-platform:{ …’ to the name.

I’ve checked on the running TX1 under

‘/proc/device-tree/tegra-camera-platform’ and have confirmed that the values inside that directory correspond to the DTS files I wrote.

Here is the output of ‘tree tegra-camera-platform’ within the /proc/device-tree which leads me to believe it is set up correctly:

tegra-camera-platform/
├── compatible
├── isp_bw_margin_pct
├── isp_peak_byte_per_pixel
├── max_lane_speed
├── max_pixel_rate
├── min_bits_per_pixel
├── modules
│   ├── module0
│   │   ├── badge
│   │   ├── drivernode0
│   │   │   ├── devname
│   │   │   ├── name
│   │   │   ├── pcl_id
│   │   │   ├── proc-device-tree
│   │   │   └── status
│   │   ├── name
│   │   ├── orientation
│   │   ├── position
│   │   └── status
│   ├── module1
│   │   ├── badge
│   │   ├── drivernode0
│   │   │   ├── devname
│   │   │   ├── name
│   │   │   ├── pcl_id
│   │   │   ├── proc-device-tree
│   │   │   └── status
│   │   ├── name
│   │   ├── orientation
│   │   ├── position
│   │   └── status
│   ├── module2
│   │   ├── badge
│   │   ├── drivernode0
│   │   │   ├── devname
│   │   │   ├── name
│   │   │   ├── pcl_id
│   │   │   ├── proc-device-tree
│   │   │   └── status
│   │   ├── name
│   │   ├── orientation
│   │   ├── position
│   │   └── status
│   └── name
├── name
├── num_csi_lanes
├── vi_bw_margin_pct
└── vi_peak_byte_per_pixel

7 directories, 41 files

Regarding ‘2’:

Is the ISP application/daemon configured using the ‘tegra-camera-platfrom’ entries within the DTS?

Thanks again for your help.

Dave

6

Hi cospan
Only the badge info have relative with ISP setting.
Can you tell how the working cache file gen from? That’s interesting.

7

Shane,

That’s a great question, I’ve been looking through my previous git commits for any changes I made to the tegra-camera-platform. As you can probably tell by my git history I haven’t been doing a great job in keeping my commits atomic. There has been a big time crunch to solve problems and for that I apologize.

I did make a lot of mistakes when I first wrote the driver/dts files and it might have been that that cache file came from not even having a badge entry filled out, or filled out incorrectly.

I just tried out these three entries for the badges:

  • uav_back_imx219: This didn't work and yielded the blue pictures
  • e3322_centerleft_A815P2: This had the same result as the ISP settings above
  • e3326_front_P5V27C: This worked! Although the image seems to have a blue tint to it

Now I’m confused…

I decided to look more into this. I have all three of the cache files for the cameras:

I wanted to capture images of all three cache files, one with the unknown, another with the e3322_centerleft_A819P2 settings and the last with the e3326_front_P5V27C. And now the configuration e3322_centerleft_A819P2 works?!

The procedure I followed was that I would copy each of those cache files to the /var/nvidia/nvcam/settings/nvcam_cache_0.bin and run nvgstcapture or a gstreamer pipeline that would capture a jpeg image.

Here are all three images for reference even though it won’t show much difference.

As a side note. Even though the e3322_centerleft_A919P2 does capture good images now it still output a stream of “LSC:patch width less than 12”

i have captured an image from camera 2 with the e3322_centerleft_A919P2 and it is ‘Green Saturated’

Here are all the images:

Unknown Cache File:

1920×1080 104 KB

e3322_centerleft_A819P2 (Working ?!)

1920×1080 198 KB

e3326_front_P5V27C

1920×1080 152 KB

Camera 2 with e3322_centerleft_A819P2 (This is looking at a different view)

1920×1080 151 KB

I can work with the e3326_front_P5V27C for now but the e3322_centerleft_A819P2 looks so much better. I would like to use that badge.

Dave

8

Hi Cospan
May I suggest to go through the tuning process to gen the right isp file? And you may need scaling partner help for it.

9

I’d like to learn how the cache binary /var/nvidia/nvcam/settings/nvcam_cache*.bin were generated from DTSI badge. Any update on the issue and the solution? My main interest is on TX2, I’ll start a new thread on TX2 board for the discussion as well.

10

Move to DTS settings and image color saturation problem with multiple CSI cameras - Jetson TX2 - NVIDIA Developer Forums