ISP Settings and 'ncam_cache'

Until recently I’ve been using an IMX219 camera sensor on the TX1 without issue. A couple of weeks ago I was attempting use gstreamer to capture video from different boards (circuit board with TX1 module) when one of the TX1 module with a camera sensor started acting strange.

On the misbehaving TX1 I could capture a raw bayer image from the camera sensor and use some userland applications to convert it to a png and the image looked fine but when I would use nvgstcapture or a gstreamer pipeline the image would come out oversaturated with one of the following: blue, red or yellow, it was like a grayscale image except instead of white as the brightest it would be blue.

I have two TX1 modules both with different rootfs, call them A and B for reference.

I can flash both TX1s with rootfs ‘A’ and they would both capture images correctly.
I can flash both TX1s with rootfs ‘B’ and they would both fail.

This proves it’s not a hardware issue (Circuit board or the TX1 Module).

If one TX1 had ‘A’ and the other had ‘B’ and I installed the same bootloader, kernel and DTB I would see the same result (‘A’ would work and ‘B’ would fail)!?

The one difference I noticed between ‘A’ and ‘B’ was if I ran ‘nvcamera-daemon’ as a foreground application and run the following gstreamer command on both:

gst-launch-1.0 -v -m nvcamerasrc sensor-id=0 fpsRange="30 30" num-buffers=1 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw, width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! jpegenc ! filesink location=/home/ubuntu/Projects/video-tester/test.jpg -e

I would see differing outputs as follows:

‘A’ (Working):

NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)

‘B’ (Not Working):

NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12LSC:patch width less than 12(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)

Note the: LSC:patch width less than 12

When I run a different pipeline to capture a video stream that above error would repeat continuously

I won’t go into the painful details about using a hex editor to view where those errors come from but I finally found something that would effect the behavior of the cameras.

in the /var/nvidia/nvcam/settings/ directory if I replaced the nvcam_cache*.bin from ‘B’ with the content of the same file within ‘A’ I wouldn’t see that error. The camera sensors would work fine and the problem is fixed.

If I delete the files and run the applications again the error would arise.

It looks like these files store the settings for the ISP for use with a camera. Unfortunately there is something going weird when the application needs to populate the camera’s cache:

Here is an excerpt from the nvcamera-daemon when I delete the cache files and re-run the jpeg capture pipeline.

...
ISP Config file did not load from cache, quering deviceNvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 170: Error: Invalid isp config attribute element: "flash.TorchLevel[0]=100.00"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 171: Error: Invalid isp config attribute element: "flash.FlashLevel[0]=180.00"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 180: Error: Invalid isp config attribute: "nr.v1.luma.Enable=TRUE"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 181: Error: Invalid isp config attribute: "nr.v1.luma.IsoThreshold=100"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 182: Error: Invalid isp config attribute: "nr.v1.luma.Slope=0.01"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 183: Error: Invalid isp config attribute: "nr.v1.luma.Threshold=0.015"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 184: Error: Invalid isp config attribute: "nr.v1.chroma.Enable=TRUE"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 185: Error: Invalid isp config attribute: "nr.v1.chroma.IsoThreshold=100"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 186: Error: Invalid isp config attribute: "nr.v1.chroma.Slope=0.40"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 187: Error: Invalid isp config attribute: "nr.v1.chroma.Threshold=0.08"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 188: Error: Invalid isp config attribute: "nr.v2.chroma.Enable=FALSE"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 189: Error: Invalid isp config attribute: "nr.v2.chroma.IsoThreshold=100"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 190: Error: Invalid isp config attribute: "nr.v2.chroma.Slope=1.0"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 191: Error: Invalid isp config attribute: "nr.v2.chroma.Threshold=0.0"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 192: Error: Invalid isp config attribute: "nr.v2.chroma.PredefinedGainResponse={0,0,0,0,0,0,1,2,3,4,4,4,4,4,4,4}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 193: Error: Invalid isp config attribute element: "ae.noiseReduction.Preview[0]={0,0,1,2,2,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 194: Error: Invalid isp config attribute element: "ae.noiseReduction.Preview[1]={0,1,1,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 195: Error: Invalid isp config attribute element: "ae.noiseReduction.Preview[2]={1,1,2,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 196: Error: Invalid isp config attribute element: "ae.noiseReduction.Preview[3]={2,2,2,3,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 197: Error: Invalid isp config attribute element: "ae.noiseReduction.Still[0]={0,0,1,2,2,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 198: Error: Invalid isp config attribute element: "ae.noiseReduction.Still[1]={0,1,1,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 199: Error: Invalid isp config attribute element: "ae.noiseReduction.Still[2]={1,1,2,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 200: Error: Invalid isp config attribute element: "ae.noiseReduction.Still[3]={2,2,2,3,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 201: Error: Invalid isp config attribute element: "ae.noiseReduction.Video[0]={0,0,1,2,2,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 202: Error: Invalid isp config attribute element: "ae.noiseReduction.Video[1]={0,1,1,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 203: Error: Invalid isp config attribute element: "ae.noiseReduction.Video[2]={1,1,2,2,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 204: Error: Invalid isp config attribute element: "ae.noiseReduction.Video[3]={2,2,2,3,3,3,3}"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 205: Error: Invalid isp config attribute: "edgeEnhance.strength.Preview=0.113"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 206: Error: Invalid isp config attribute: "edgeEnhance.strength.Still=0.113"
NvCameraIspConfigFileLoad: Config file "sensor.cfg" Line 207: Error: Invalid isp config attribute: "edgeEnhance.strength.Video=0.113"
LSC:patch width less than 12
...

I’ve been searching around for a ‘sensor.cfg’ file but couldn’t find it.

I would think that the dts file for the camera would be the thing used to populate this cache file but it doesn’t seem as though the changes I make within the DTS file fixes this problem.

How can I configure the ISPs correctly whenever the cache needs to be rebuilt?

I don’t have access to the NVIDIA partners program so all of these files are opaque to me.

Thanks in advance for any help,

Dave

Hi cospan

Does this sensor module from Leaopard or some others vendor? I guess you can contact them to get the isp config file for you problem.

Shane,

Thanks for the feedback! No the driver is not from a vendor, I wrote it.

It would like to have access to adapt this sensor but barring that I would be happy if someone on the inside would let me know what values I have to set in order to get the nvcam_cach*.bin to build correctly.

If you would like a reference to the driver, mode table and dtsi file for the camera I’ve put them here.

Camera DTSI File: https://raw.githubusercontent.com/CospanDesign/nvidia-tx1-kernel/mit-uav-revb/arch/arm64/boot/dts/tegra210-platforms/tegra210-camera-mit-uav-revb.dtsi
Driver: https://raw.githubusercontent.com/CospanDesign/nvidia-tx1-kernel/mit-uav-revb/drivers/media/i2c/imx219.c
Mode Tables: https://raw.githubusercontent.com/CospanDesign/nvidia-tx1-kernel/mit-uav-revb/drivers/media/i2c/imx219_mode_tbls.h

As a note I have three cameras attached to the board and I use a GPIO I2C multiplexer to access different cameras.

If possible perhaps someone could let me know what values on the dtsi file I could change in order to make this work.

Hi cospan
You can try to modify the badge info exactly the same like tegra210-camera-e3322-a00.dtsi to check if it can load the ISP setting for imx219. Otherwise you need scaling partner to help to tuning to gen an isp file.

module1 {
457				badge = "e3322_centerleft_A815P2";
458				position = "centerleft";
459				orientation = "1";
460				drivernode0 {
461					pcl_id = "v4l2_sensor";
462					devname = "imx219 31-0010";
463					proc-device-tree = "/proc/device-tree/host1x/i2c@546c0000/tca9548@77/i2c@1/imx219_b@10";
464				};
465			};
466			module2 {
467				badge = "e3322_centerleft_A815P2";
468				position = "centerright";
469				orientation = "1";
470				drivernode0 {
471					pcl_id = "v4l2_sensor";
472					devname = "imx219 32-0010";
473					proc-device-tree = "/proc/device-tree/host1x/i2c@546c0000/tca9548@77/i2c@2/imx219_c@10";
474				};
475			};
476			module3 {
477				badge = "e3322_centerleft_A815P2";
478				position = "topleft";
479				orientation = "1";
480				drivernode0 {
481					pcl_id = "v4l2_sensor";
482					devname = "imx219 33-0010";
483					proc-device-tree = "/proc/device-tree/host1x/i2c@546c0000/tca9548@77/i2c@3/imx219_d@10";
484				};
485			};

Shane,

Thanks for looking into this, that was a good idea, I should have thought of it.

I tried it out but no good, the nvcamera-daemon rebuilt the cache with bad values again. I grabbed the previous (working) nvcam_cache_0.bin and the one that was built and can see the difference but everything is in a binary format and it wouldn’t be trivial to reverse engineer it.

I ran an experiment:

I wanted to see how the ‘nvcam_cache_*.bin’ files were generated relative to the settings within the DTS:tegra_camera_platform entries.

Now I have the following three nvcam_cache0.bin files:

  • The old, working version
  • Version using my original values
  • Version with values from E3322 board values

I ran a diff of each of the files and came out with this result:

1 and 2 are different
1 and 3 are different
2 and 3 are the same?!

I would have thought that the nvcam_cache files generated from two different tegra-camera-platform groups would have yielded different cache files. There is something I’m not understanding.

From this I can conclude one of two things is happening:

  1. There is something wrong with the way I'm writing my DTS file
  2. The userland ISP application is independent of the 'tegra-camera-platform' entry within DTS file.

Regarding ‘1’:

I looked in other DTS files within the tegra210-platform directory in order to determine if there is something else that I should be putting inside there. I’ve looked inside ‘tegra210-platform/tegra210-jetson-cv-camera-modules.dtsi’ file and made sure that my ‘tegra-camera-platform’ looks exactly the same, even adding in the ‘tcp:tegra-camera-platform:{ …’ to the name.

I’ve checked on the running TX1 under

‘/proc/device-tree/tegra-camera-platform’ and have confirmed that the values inside that directory correspond to the DTS files I wrote.

Here is the output of ‘tree tegra-camera-platform’ within the /proc/device-tree which leads me to believe it is set up correctly:

tegra-camera-platform/
├── compatible
├── isp_bw_margin_pct
├── isp_peak_byte_per_pixel
├── max_lane_speed
├── max_pixel_rate
├── min_bits_per_pixel
├── modules
│   ├── module0
│   │   ├── badge
│   │   ├── drivernode0
│   │   │   ├── devname
│   │   │   ├── name
│   │   │   ├── pcl_id
│   │   │   ├── proc-device-tree
│   │   │   └── status
│   │   ├── name
│   │   ├── orientation
│   │   ├── position
│   │   └── status
│   ├── module1
│   │   ├── badge
│   │   ├── drivernode0
│   │   │   ├── devname
│   │   │   ├── name
│   │   │   ├── pcl_id
│   │   │   ├── proc-device-tree
│   │   │   └── status
│   │   ├── name
│   │   ├── orientation
│   │   ├── position
│   │   └── status
│   ├── module2
│   │   ├── badge
│   │   ├── drivernode0
│   │   │   ├── devname
│   │   │   ├── name
│   │   │   ├── pcl_id
│   │   │   ├── proc-device-tree
│   │   │   └── status
│   │   ├── name
│   │   ├── orientation
│   │   ├── position
│   │   └── status
│   └── name
├── name
├── num_csi_lanes
├── vi_bw_margin_pct
└── vi_peak_byte_per_pixel

7 directories, 41 files

Regarding ‘2’:

Is the ISP application/daemon configured using the ‘tegra-camera-platfrom’ entries within the DTS?

Thanks again for your help.

Dave

Hi cospan
Only the badge info have relative with ISP setting.
Can you tell how the working cache file gen from? That’s interesting.

Shane,

That’s a great question, I’ve been looking through my previous git commits for any changes I made to the tegra-camera-platform. As you can probably tell by my git history I haven’t been doing a great job in keeping my commits atomic. There has been a big time crunch to solve problems and for that I apologize.

I did make a lot of mistakes when I first wrote the driver/dts files and it might have been that that cache file came from not even having a badge entry filled out, or filled out incorrectly.

I just tried out these three entries for the badges:

  • uav_back_imx219: This didn't work and yielded the blue pictures
  • e3322_centerleft_A815P2: This had the same result as the ISP settings above
  • e3326_front_P5V27C: This worked! Although the image seems to have a blue tint to it

Now I’m confused…

I decided to look more into this. I have all three of the cache files for the cameras:

I wanted to capture images of all three cache files, one with the unknown, another with the e3322_centerleft_A819P2 settings and the last with the e3326_front_P5V27C. And now the configuration e3322_centerleft_A819P2 works?!

The procedure I followed was that I would copy each of those cache files to the /var/nvidia/nvcam/settings/nvcam_cache_0.bin and run nvgstcapture or a gstreamer pipeline that would capture a jpeg image.

Here are all three images for reference even though it won’t show much difference.

As a side note. Even though the e3322_centerleft_A919P2 does capture good images now it still output a stream of “LSC:patch width less than 12”

i have captured an image from camera 2 with the e3322_centerleft_A919P2 and it is ‘Green Saturated’

Here are all the images:

Unknown Cache File:

e3322_centerleft_A819P2 (Working ?!)

e3326_front_P5V27C

Camera 2 with e3322_centerleft_A819P2 (This is looking at a different view)

I can work with the e3326_front_P5V27C for now but the e3322_centerleft_A819P2 looks so much better. I would like to use that badge.

Dave

Hi Cospan
May I suggest to go through the tuning process to gen the right isp file? And you may need scaling partner help for it.

I’d like to learn how the cache binary /var/nvidia/nvcam/settings/nvcam_cache*.bin were generated from DTSI badge. Any update on the issue and the solution? My main interest is on TX2, I’ll start a new thread on TX2 board for the discussion as well.

Move to https://devtalk.nvidia.com/default/topic/1026723/