Add new sensor driver with Orin NX (t234)

Hi,
I try to add a new sensor driver and got confused with the new source and dt source files.

I read this :
https://docs.nvidia.com/jetson/archives/r35.5.0/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html#device-registration

1.It refers to t19x and not t234.
2. I didn’t find where this appers (on which file)

imx185_cam0: imx185_a@1a {
status = “disabled”;
};

So what is the full process for adding a new sensor driver based on the imx185 existing driver?

  1. Copy the imx185 source code files and modify them for my new driver.
  2. which of the following:

2.1. Copy the file containing the following node:

tegra-camera-platform {
compatible = “nvidia, tegra-camera-platform”;
modules {
module0 {
badge = “imx185_bottom_liimx185”;
position = “bottom”;
orientation = “0”;
drivernode0 {
pcl_id = “v4l2_sensor”;
devname = “imx185 30-001a”;
proc-device-tree =
“/proc/device-tree/i2c@3180000/tca9546@70/i2c@0/imx185_a@1a”;
};
};
};
};

and overwrite it with my own tegra-camera-platform tree node

OR

2.2. Locate the file containing the tegra-camera-pltaform for the t234 chip and add a new module node (aka module1) for my new device?

  1. How to add the .dtsi from step 2 to the general device tree?
  • What is the equivalent of the following node for my new sensor:

imx185_cam0: imx185_a@1a {
status = “disabled”;
};

(its not part of the imx185 .dtsi)

  • the overlay in the guide is regarding t19x models. Are the instructions the same for t234?
  1. In order to add the kernel config: CONFIG_VIDEO camera>=m
    where the camera string is taken from?
    does it takes it from here:

.driver = {
.name = “imx185”,
.owner = THIS_MODULE,
.of_match_table = of_match_ptr(imx185_of_match),
},

thanks

hello BSP_User,

>>Q1
please works with t234, actually, you should refer to Orin NX’s device tree sources under $public_sources/kernel_src/hardware/nvidia/platform/t23x/p3768/ for development.

>>Q2
as you may know, it’s later include device tree (or, overlays) to update the status to enable the node.
here’s an example,
$public_sources/kernel_src/hardware/nvidia/platform/t23x/p3768/kernel-dts/cvb/tegra234-p3768-camera-rbpcv3-imx477.dtsi

>>Q3
you’ll see a main device tree to include all camera device trees.
$public_sources/kernel_src/hardware/nvidia/platform/t23x/p3768/kernel-dts/cvb/tegra234-p3768-0000-a0.dtsi

>>Q4
please refer to developer guide, To Create and Apply a DTB Overlay File.
you may see-also kernel implementation, it’s a configuration within $public_sources/kernel_src/kernel/nvidia/drivers/media/i2c/Makefile

Hi,
Thank you for your help.
First I wish to emphasize that this is my first experience with adding sensor/new device support , therefore I’m relying heavily on the dev guide and its examples. Since they are not suitable for t234 I’m asking you more questions to understand.

Q1:
On the one hand the camera developing guide refers to imx185 sensor as a base reference, while
on the other hand there is no imx185 device tree source file under $public_sources/kernel_src/hardware/nvidia/platform/t23x/p3768/

so what to do?

Q2-Q3:
I’m asking specifically regarding this string:

imx185_cam0: imx185_a@1a

  • 1a is the I2C slave address
  • what is imx185_a ? just a name? or a specific node’s name?
  • what is imx185_cam0 ? just a name? or a specific node’s name? and where to put it?

I think my missing piece here is related to distinguish between two separate nodes of the same device:

  • one needs to be under the I2C node
  • other needs to be under the tegra-camera-platform node?
    If so, the guide states how and where to put the tegra-camera-platform sub node but how I know where to put the I2C sub node?

It will be much helpful to get some elaboration here since I’m a little bit confused I think.

Q4:
To make sure I understand, the following line for example:

obj-$(CONFIG_I2C_IOEXPANDER_SER_MAX9295) += max9295.o

means that if i add: CONFIG_I2C_IOEXPANDER_SER_MAX9295=m/y in the defconfig file
than it will compile the suitable sources for max9295.o upon kernel build?

hello BSP_User,

>>Q1
note, I’ve point-out t23x/p3768/ since you’re developing based-on Orin NX.
due to different camera connector, there’s no IMX185 device tree for Orin NX/Nano series.
so… please refer to IMX185 reference driver with Jetson AGX Orin series. (i.e. t23x/concord/)
for instance,
you may check below two device tree for reference.
(1) it’s device tree with IMX185 sensor specific settings.
$public_sources/kernel_src/hardware/nvidia/platform/t23x/common/kernel-dts/t234-common-modules/tegra234-camera-imx185-a00.dtsi
(2) here’s camera control specific settings.
$public_sources/kernel_src/hardware/nvidia/platform/t23x/concord/kernel-dts/cvb/tegra234-p3737-0000-camera-imx185-a00.dtsi

>>Q2, Q3
yes, that’s just an unique string. usually, we used CSI-port and I2C slave address to specify a camera node.
in this example, imx185_a@1a,
it means it’s IMX185 for using CSI port-A, which is under I2C at 0x1a.

>>Q4
yes, it’s just a variable,
you may use CONFIG_I2C_IOEXPANDER_SER_MAX9295=m to build as a module. (*.ko)
or, CONFIG_I2C_IOEXPANDER_SER_MAX9295=y to always build-in the driver with the kernel image.

Hi @JerryChang ,
By your permission I wish to sum it up to make sure I understood. I have a question on step 3.
Adding support for “my_new_sensor”:

  1. Step 1: I copy the file:

$public_sources/kernel_src/hardware/nvidia/platform/t23x/common/kernel-dts/t234-common-modules/tegra234-camera-imx185-a00.dtsi

into:

$public_sources/kernel_src/hardware/nvidia/platform/t23x/p3768/

and rename it to indicate its for “my_new_sensor”. So the result is the new file:

$public_sources/kernel_src/hardware/nvidia/platform/t23x/p3768/tegra234-camera-my_new_sensor.dtsi

Explanation:

  • You said I need to work in

$public_sources/kernel_src/hardware/nvidia/platform/t23x/p3768/

  • I use imx185 for reference as the dev guide instruct
  1. Step 2: Modify the new “my_new_sensor” .dtsi file to contain the new sensor values. I modify this:

$public_sources/kernel_src/hardware/nvidia/platform/t23x/p3768/tegra234-camera-my_new_sensor.dtsi

(including modifying the “tegra-camera-platform” node as the devguide instructs)

  1. Step 3: here I still don’t understand.
    From my understanding so far, there are two .dtsi files needed to be copied/modified.
  • The one mentioned in step 1:

$public_sources/kernel_src/hardware/nvidia/platform/t23x/common/kernel-dts/t234-common-modules/tegra234-camera-imx185-a00.dtsi

  • The one contains the “imx185_cam0” string.

Now:
I wish to locate the file containing “imx185_cam0” and modify this node to “my_new_sensor_cam0”. (it appears in a snapshot in the devguide)

My question is:
3.1. where to do it? where is this node should be and where is the reference.
From your answer I understand its related to the i2c tree. Do I copy & modify it (like in step 1) or just modify it and replace the node’s name from “imx185_cam0” to “my_new_sensor_cam0”

3.2. The confusing part is that in addition to “imx185_cam0” which is related to i2c (that i need to modify for my new sensor) there is additional node related to i2c but there is no “imx185_cam0” in it. The node is: i2c@3180000->tca9546@70->i2c@0->imx185_a@1a
why there are 2 different nodes related to i2c and how I modify both of them for my new sensor?

for example, in my new sensor .dtsi it will be: " i2c@3180000->tca9546@70->i2c@0->my_new_sensor@[i2c slave addr]" ?

  1. Step 4: Create dt overlay as the devguide instructs and use the Jetson-IO tool

Thank you for your patience.

hello BSP_User,

besides modify the node’s name to your my_new_sensor_cam0, please also refer to Port Binding section to update the port definition accordingly.
you may also have examination by… To verify the port binding result.

in which file i put it?
the “my_new_sensor_cam0” node

hello BSP_User,

please check the reference drivers for development.
you may either check AGX Orin’s IMX185 sensor driver, or, you may check IMX219 or IMX477 on Orin NX/Nano.

OK, I think I saw it.
For learning purposes, why there are two different .dtsi files for the same camera:

hardware\nvidia\platform\t23x\p3768\kernel-dts\cvb\tegra234-p3768-camera-rbpcv2-imx219.dtsi

and

hardware\nvidia\platform\t23x\p3768\kernel-dts\cvb\tegra234-camera-rbpcv2-imx219.dtsi

Is this related to some standard or “the right way” of doing things?

that’s same as my explanations in previous comment #5.
there’re one device tree with sensor specific settings, the other device tree for camera control (and, Jetson board) specific settings.

Thank you for your help.

Is it ok by you the I’ll edit my changes and post here to make sure I did everything the way needed?

hello BSP_User,

you may refer to previous comment #7 to verify your driver first.
please see-also developer guide. Approaches for Validating and Testing the V4L2 Driver.

Thank for your reply.
I have a question please regarding the DTB Overlay (the answer not appears in the dev guide)

I created two new files:
a) tegra234-camera-new-sensor.dtsi (sensor device tree)
b) tegra234-p3767-camera-new-sensor.dts (dt overlay)
(like the imx477 reference there is no include statement between them)

  1. How I compile both files into one .dtbo? (I need to “attach” both files even tough there is no include between them)
    For exmaple:
    dtc -O dtb -o new_sensor.dtbo -@ tegra234-p3767-camera-new-sensor.dts

  2. For sanity check I executed:

dtc -O dtb -o new_sensor.dtbo -@ tegra234-p3767-camera-imx477-dual.dts

and got:

Error: tegra234-p3767-camera-imx477-dual.dts:13.1-9 syntax error
FATAL ERROR: Unable to parse input tree

What seems to be the problem?

Thanks

hello BSP_User,

it looks an incorrect commands.
please execute the following to convert the DTS into a new DTB file.
i.e. $ sudo dtc -I dts -O dtb -o new-output.dtb temp.dts

furthermore,
please check the configuration file /boot/extlinux/extlinux.conf
it’s FDT entry for device tree blob; OVERLAYS for device tree overlay.
please note that, You can specify FDT alone. You can specify FDT + OVERLAYS. You cannot do only OVERLAYS though.

Thanks for your answer.
I have additional questions by your permissions:

  1. when I convert the dts to dtb, there is no connection between the .dtsi with all the sensor/csi/vi properties-values pairs and the .dts which contain the fragment property.
    There is no include statement between them.

So how the compiler knows to take both the .dts which contains the fragment and overlay properties and the .dtsi which contain the sensor/csi/vi properties and combine them into one binary?

  1. My sensor is actually a deserializer that generates pattern. using i2c tools like i2cdetect I saw that the sensor is on i2c bus num 9. Where in the .dtsi I mention this detail? (I wrote the slave address but didn’t write the I2C bus number)

hello BSP_User,

>>Q1
please refer to developer guide, Building the Kernel.
you may try to build from sources, it’s a bad practice to use dtc utility to convert dtsi file into device tree blob.

>>Q2
please refer to IMX390 sensor driver, it’s a reference driver for using GMSL SerDes chip.
you may see-also developer guide, Jetson Virtual Channel with GMSL Camera Framework.

Hi @JerryChang,
I apologise but I didn’t understand you.
I wish to get back to the start just to make sure we both aligned and save confusion.

  1. My setup is: FPD Link III de-serializer which acts as a pattern generator connected to the Orin NX via 2x MIPI CSI.
  2. I use the sensor programming guide. The problem is that its outdated. It doesn’t contains explanations nor examples for Jetsons based on t234 chips (like Orin family).
  3. Right now I’m troubling with the device tree setup in order to get the expected Media-ctl output (verify the ports binding of the des->csi->vi)

What I did so far based on the programming guide and your answers:
a. Created a single .dtsi for my “new sensor” (fpd link des). I filled it based on the imx477 reference as you mentioned.

Now In order to execute the media-ctl to verify the ports binding I wish to perform the Device Registration as mentioned here:
https://docs.nvidia.com/jetson/archives/r35.5.0/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html#device-registration
It says I have several options to register my .dtsi in the kernel device tree:

  1. Use device tree overlay
  2. Using the Jetson-IO Tool
  3. Using the Main Platform Device Tree File

If I want to choose option 1 the guide says:

  1. Add your .dtsi file to the camera configuration .dtsi file.
  • I don’t know what is the camera configuration .dtsi in my case (t234, orin nx, “new sensor”)
  1. Compile the .dts file to generate a .dtbo file. Move the .dtbo file to flash_folder/kernel/dtb/ before flashing.
  • Here it states to compile the .dts created as overlay but it doesn’t say what to with my .dtsi for my new sensor (we have wo new files - .dtsi for my sensor and .dts for overlay)

So the instructions in this options are not cleared.

If I want to choose option 3 the guide says:

  1. Locate and edit the .dtsi file:

/hardware/nvidia/platform/t19x/galen/kernel-dts/tegra194-p2888-0001-p2822-0000.dtsi

  1. Remove the following line from the file:

include “tegra194-camera-plugin-manager.dtsi”

  1. Replace the following line:

include “common/tegra194-p2822-camera-modules.dtsi”

With an #include statement specifying the .dtsi file for your new device.
4. Save and close the file.

  • As I said, the files mentioned here are not relevant for my case. In addition there is no plugin manager in systems using UEFI. So the question here is which file to edit.

What I did:

  • Edited tegra234-p3768-0000-a0.dtsi by removing the inclusions of imx477 and imx219 and adding inclusion for my .dtsi
  • built the kernel dtbs only (not the entire kernel from source but just the dtbs)
  • moved the final dtb to the target

The problem is that this solution didn’t work for me as well. It produces two problems:

  1. The final generated dtb, for some reason, ignores the vi/csi nodes from my new_sensor.dtsi but takes other nodes from it.
  2. The Media-Ctl output is not as expected.

So to conclude I’m stuck since I find it hard to “guess” the correct operations needed to build the proper device tree for my case. The devguide is outdated and its my first time developing new sensor driver.

hello BSP_User,

Xavier and Orin series they’re sharing the same VI driver, you may still refer to developer guide even though it did not mentioned t23x.
and… it’s always suggest to check reference drivers for deployment.

I followed the dev-guide and the reference drivers.
I wrote my own .dtsi for the new device and a kernel module.
My module’s probe function is not executing (probably something wrong with my device registration/identification from the device tree)

I’m kinda stuck. Is it possible to get you help regarding this subject?
My setup is very simple:
FPD Link de-serializer → Orin NX

hello BSP_User,

please refer to Device Registration section.
it’s suggest to work with [Using the Main Platform Device Tree File] for force system booting up with your customize module.