Custom camera driver for TX2 NX

Can someone clarify what is the recommended approach for adding additional camera support to TX2 NX besides the few IMX sensors that are built in the kernel? Specifically, should I use LKM, kernel overlay with or without jetson-io tool, full kernel build from source, patch files, etc.? I see many different approaches on the forums and documentation but no tutorials from beginning to end and I am missing something. This seems like a topic that would benefit many people as cameras are constantly changing and computer vision is fundamental to the reason the Jetson platform is so popular.

Carrier board: Seeed A203v2
Module: Jetson TX2 NX
Jetpack: 4.6 (32.6.1)
Host pc: Linux 18.04
MIPI CSI Camera: e-consystems e-CAM131_CUNX with AR1335 sensor or Arducam IMX477

For reference, I have successfully flashed the board (with custom .dtb kernel file from Seeed) and tested an IMX219 camera without issue. I built a custom .dts file for the AR1335 and I thought I added it to the device tree but the camera was not recognized after following the Kernel Customization and Camera Development documentation. Apparently I missed a step. So can someone explain what to do after I create the .dtsi file that looks like the example " IMX185 V4L2 sensor driver" in the “Sensor Software Driver Programming” documentation?

Suggest following the programing guide to implement the driver and device tree.

@ShaneCCC, first of all, I appreciate your prompt response but admittedly, I hoped for some more insight. I did follow the documentation as far as I can tell… but there are a lot of details missing and it didn’t work for me. Maybe some more details on my methods would help uncover the issue. At the very least, I just want to know the “best practice” for adding a new camera on a TX2 NX.
These are the steps I followed directly from the “Sensor Software Driver Programming” documentation:

  1. Create a tegra-camera-platform device tree node in the kernel source tree (.dtsi file)
  2. Verify port binding with sudo media-ctl -p -d /dev/media0 (what? it isn’t flashed yet…not sure how to do this step here…)
  3. Loadable Kernel Module (LKM). Oh this sounds fantastic! But how do I do it? Docs say by placing CONFIG_VIDEO =m in kernel configuration file (I assume this is source/kernel/kernel-4.9/arch/arm/configs/tegra_defconfig). Okay did that… now what?
  4. Rebuild the kernel and the sensor driver (again… how? build the entire kernel from source?). I built the kernel source how the other documentation says but ar1335.ko is nowhere to be found… that is too bad. This method sounded promising for debugging.
  5. Device registration. Should I use plugin manager, Jetson-IO tool, or Main Platform Device Tree File for TX2 NX? I have read that Jetson-IO tool only works with certain platforms and plugin manager is not the preferred method anymore. I tried to create a .dtbo file but my include statements gave errors. That left me with Main Platform Device Tree File method. So I changed the .dtsi file to include my camera .dtsi file and comment out camera-plugin-manager.dtsi
  6. That’s it for instruction in the docs. So I assume I need to build the kernel and flash the jetson for any of these .dtsi and config file changes to take effect? I tried but the jetson image and device tree appears unchanged.

I have since tried using JP4.6.2 (32.7.2) with no success either. This is how I built the kernel and flashed:

  1. Downloaded and untarred public_sources.tbz2 (also ran alternative method of ./source_sync.sh -k tegra-l4t-r32.7.1 to get sources folder)
  2. Unpacked kernel_src.tbz2 to get hardware/ and kernel/
  3. Made modifications as follows (I also did the same mods by creating a patch file if you want me to share that)
  • added tegra186-camera-ar1335.dtsi and tegra186-camera-lanai-ar1335.dtsi to sources/hardware/nvidia/platform/t18x/lanai/kernel-dts/common
  • Updated sources/hardware/nvidia/platform/t18x/lanai/kernel-dts/tegra186-p3636-0001-p3509-0000-a01.dts
  • Added driver sources/kernel/nvidia/drivers/media/ar1335.c
  • Added sources/kernel/nvidia/include/media/ar1335.h
  • Added CONFIG_VIDEO ar1335=y to tegra_defconfig
  1. Ran various make -Ccommands per docs to create dtb and Image file
  2. Tried several variants of flashing sudo ./flash.sh -r -k kernel-dtb -d kernel/dtb/tegra186-p3636-0001-p3509-0000-a01.dtb jetson-xavier-nx-devkit-tx2-nx nvme0n1p1
  3. The result? Only the imx219 camera works (which is the default config)…

Any details would be greatly appreciated… I might be bald soon from pulling my hair out…

  1. Did you add ar1335.c to …/kernel/nvidia/drivers/media/i2c/Kconfig?
  2. In the $TEGRA_KERNEL_OUT/drivers/media/i2c/
  3. Using main platform device tree would be easy for your case.
  4. Did you replace your dtb and Image at …/Linux_for_Tegra/kernel/ before flash? How do you verify it?
1 Like

These comments were very helpful, thank you.

  1. I did not originally add ar1335.c to Kconfig. After adding this, I successfully ran these commands:
    make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT tegra_defconfig
    make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT menuconfig
    which allowed me to select the ar1335 under Device Drivers -><*> Multimedia support ->NVIDIA overlay Encoders, decoders, sensors and other helper chips -><*M> AR1335 camera sensor support. For the sake of documentation, this was followed by these 4 commands:
    make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j8 Image
    make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j8 dtbs
    make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j8 modules
    make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT modules_install INSTALL_MOD_PATH=$KERNEL_MODULES_OUT
  2. I used the main platform device tree method and modified these files under source/hardware/nvidia/platform/t18x/lanai/kernel-dts/
  • tegra186-p3636-0002-p3509-0000-a01.dts and tegra186-p3636-0001-p3509-0000-a01.dts
    • Remove: #include "common/tegra186-camera-lanai-rbpcv2-imx219.dtsi
    • Add: #include "common/tegra186-camera-lanai-ar1335.dtsi"
  • tegra186-p3636-0002-lanai.dts
    • Remove: #include "common/tegra186-p3636-plugin-manager.dtsi"
    • Add: #include "common/tegra186-camera-lanai-ar1335.dtsi"
  1. I did replace dtb and Image. I verified based on the timestamp as well as copying the file and running dtc -I dtb -O dts -o modified.dts tegra186-p3636-0001-p3509-0000-a01.dtb to check the source code.

Another error on my part was using the “-r” option during flash. This was reusing the old image. After using this command: sudo ./flash.sh -k kernel-dtb -d kernel/dtb/tegra186-p3636-0001-p3509-0000-a01.dtb jetson-xavier-nx-devkit-tx2-nx nvme0n1p1, I was able to properly flash the Jetson TX2 NX. I verified by logging in to the Jetson and running dtc -I fs -O dts -o extracted.dts /proc/device-tree and searching for “ar1335” in the extracted.dts file and it looks correct. As recommended, I also created a LABEL backup in /boot/extlinux/extlinux.conf to boot from the backup image in case something goes wrong with my custom kernel.

I still can’t see /dev/video0 on the device but at least I am able to modify drivers and flash so it is probably an error in my .dtsi or driver files which I will explore next. At least I can debug now :)

Check dmesg | grep -i ar1335 to check if sensor driver loaded and probe

1 Like

Thanks again for taking the time to help me troubleshoot this.

I checked dmesg and unfortunately ar1335 wasn’t found. I noticed on my host pc that <top>/kernel/kernel-4.9/arch/arm64/configs/tegra_defconfig was missing CONFIG_VIDEO_AR1335=m so i added that and tried again. The
/lib/modules/4.9.253/kernel/drivers/media/i2c/ folder does not contain ar1335.ko and no ar1335.o anywhere after CMAKE but it does have a few others, including imx477.ko. Is there a Makefile that needs to be manually edited somewhere?
Since I have the imx477, I tried to imx477.ko to /lib/modules/4.9.253-tegra/kernel/drivers/mdeia/i2c on the jetson but i get an error when running sudo insmod imx477.ko to do the LKM method. The error is: “could not insert module imx477.ko: Invalid module format”. Some of the other modules load just fine so must be an issue with that one… I was hoping to get lucky.

If you build as LKM you need to probe after boot to system.

sudo modprobe ar1335

I get FATAL: Module ar1335 not found in directory /lib/modules/4.9.253-tegra when I run sudo modprobe ar1335.

I don’t get any errors when I run sudo modprobe imx477 but it still has the same error when I sudo insmod imx477.ko (I think that specific driver needs to be modified on my end… I just grabbed the existing file to try)

You need copy from your build tree it to TX2

I copied the /lib/modules/4.9.253 folder over from host to Jetson and renamed it to 4.9.253-tegra after renaming the original to 4.9.253-tegra-backup. Hopefully that is okay.

I still get the same modprobe error that module ar1335 is not found in the directory. I suspect the Makefile or something else is missing because it isn’t building the ar1335.ko or ar1335.o files.

You can modify below to add “obj-m += ar1335.o” to build the ko

…/kernel/nvidia/drivers/media/i2c/Kconfig

I actually had to add obj-$(CONFIG_VIDEO_AR1335) += ar1335.o to the Makefile, not Kconfig in that folder but that definitely fixed some of my issues. I finally got some ar1335.c and .h file errors (meaning it was building the driver) after adding that to the Makefile and running make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j8 modules. I followed this with the rest of the build, flash, and moved .dtb, Image, and lib/modules over to the Jetson.

Now when I run dmesg | grep -i ar1335 on the jetson, I get the error “ar1335: disagrees about version of symbol module_layout” and sudo modprobe ar1335 gives me “ERROR: could not insert ‘ar1335’: Exec format error”. I suspect I need to start over and run make clean and do a fresh build on my folders in case some older kernel dependencies / files are still lingering… I will work on that over the weekend and report back. Thanks for your continued guidance @ShaneCCC.

Okay, I think this is mostly solved at this point. I still don’t have a working camera but it is recognized and loading the driver. My previous errors “Exec format error” were due to a JetPack version mismatch. I saw this error in several places with dmesg. I originally flashed JP4.6 (32.6.1) but my last kernel build from source was JP4.6.2 (32.7.2). After creating a clean folder on host PC and rebuilding from source using 32.6.1 files, I was able to recognize the cameras with my custom kernel. Now my error is related to driver issues, which I expected because I copied a lot from drivers for other jetpack/carrier-board combinations.

Latest errors:
dmesg | grep -i ar1335
[ 3.422548] Modules linked in: ar1335(+) imx219(-) spidev(+) nvgpu ip_tables x_tables
[ 3.422614] [] ar1335_probe+0xd0/0x1260 [ar1335]
[ 3.422645] [] ar1335_i2c_driver_init+0x18/0x30 [ar1335]
[ 3.422670] ar1335 9-0042: pwdn gpios not in DT
[ 3.422673] ar1335 9-0042: unable to get platform data
[ 3.422681] ar1335: probe of 9-0042 failed with error -14
[ 3.422761] ar1335 10-0042: ar1335_probe[2863]:GPIO reset Fail, err:-16
[ 3.422765] ar1335: probe of 10-0042 failed with error -22
[ 3.479196] Modules linked in: cfg80211 ar1335 imx219(-) spidev(+) nvgpu ip_tables x_tables
[ 3.531645] Modules linked in: cfg80211 ar1335 spidev(+) nvgpu ip_tables x_tables