Auvidea J120 and Intel RealSense D435

Hi,

The latest firmware for J120 [url]http://www.auvidea-pcb.com/firmware/tx2/1.6/auvidea-kernel-J90-J120-v1.6.zip[/url] provides a pre-built kernal image that needs to be replaced when flashing TX2 using JetPack.

Here is my problem, I am also planning to attach Intel RealSense D435 camera to J120’s USB 3.0 port. Now the kernel needs to be patched and built manually to fully support the camera see [url]https://www.jetsonhacks.com/2018/07/10/librealsense-update-nvidia-jetson-tx-dev-kits/[/url]. This means I cannot use the pre-built kernel provided by Auvidea and simply replace it while flashing. Is there a way to manually apply all the patches for J120 while I am building the kernel with patches for the D435 camera?

I couldn’t find instructions on Auvidea web site for applying the patches to kernel manually. can somebody help me in this?

Thanks
Nouman

Hi nouman.hanif,

Have you resolved the patch applying issue? Or contact with Auvidea team to get the support?

Thanks

Hi kayccc,

Thanks for reaching out. I raised a ticket with Auvidea support more than a week ago but haven’t had any word from them. I am still stuck unfortunately. I am new to the fascinating world of Jetson and Linux so I am struggling at the moment. I tried going through readme files accompanying Auvidea firmwares but to be honest these are very vague and lack crucial details.

I am all set to build the kernel now but still no idea how I can ‘plug’ the patches from Auvidea. Should I use their dts and dtsi files, put them somewhere so that when I am building the kernel, the build process picks these up and include in the build (or create dtbs)?

Secondly, I am not very clear on flashing the newly built kernel. Should I be using flash.sh script or the JetPack installer? I have already flashed my TX2 using the pre-built kernel Auvidea provided via JetPack installer but without the patches for D435 camera obviously.

Now I am planning to patch kernel source for D435 camera support and build plus somehow include dts/dtsi files from Auvidea in there and once ready, just flash the kernel and dtbs using flash.sh. I got dts/dtsi files from source folder of Auvidea’s 1.6 firmware package. I already have flashed TX2 with pre-built kernel from this same package.

Thanks

Hi nouman.hanif,

For flashing built kernel, you could use flash.sh script, please refer to Flashing and Booting the Target Device section from NVIDIA Tegra Linux Driver Package Development Guide - [url]https://developer.nvidia.com/embedded/dlc/l4t-documentation-28-2-ga[/url]

Thanks

Hi kaycc,

My problem is not flashing. I want to know whats the best way to include both patches for D435 camera and dts files for Auvidea J120 in kernel source before building? I want to build a kernel that can support D435 camera and Auvidea J120 carrier board on TX2?

Thanks

I cannot give any details, I do not know what the changes are for Auvidea or the camera. You may find it useful though to understand that although device tree files exist in kernel source, that the kernel edits and device tree edits are actually separate. You could copy a new kernel in (or a module if adding something in module format), and then edit the device tree later (the device tree needs to be flashed in more recent releases…once upon a time it was a simple file copy). Shell scripts designed to make it easy to add kernel modifications can also make it difficult to understand if you are really editing kernel source or device tree.

Auvidea will have its own device tree edits (and you’d have to ask Auvidea what device tree to use), a camera might have device tree content as well (and author of the driver the camera uses would probably be able to tell you what device tree changes are related). Kernel source edits, if to “.dtsi”, or “.dts” files, are not really kernel source edits, but are instead are a way of packaging device tree along with drivers which will use the tree.

Think of the device tree as a way to give uniform naming to hardware register setup which differs depending on manufacturer…the driver can use the uniform naming scheme regardless of who makes the hardware. An actual kernel source edit might change what a driver pays attention to in the tree, e.g., when back-porting or adding support for some concept the driver previously had no concept of.

Hi linuxdev,

Thanks for the explanation. I read the link you referred in my other post about building and flashing kernel and it was very helpful.

I will try to separately share all the information I have from Auvidea carrier board and Intel D435 camera manufacturers

Auvidea

They provided kernel image, dtbs, packages (kernel_supplementz.tbz2). I used these and flashed TX2 via Jetpack (replaced above files just before the jetpack wizard was ready to flash). So I have a TX2 in running state using their package.

D435 Camera
Intel provided edits to .config and patches to kernel source (.patch files that only patched .h and .c files no dts or dtsi was patched). I was initially editing the config using a script and the kernel build was failing. Then I used nconfig that you suggested and now the kernel is building with the config edits and patches included.

Now, I don’t know why Auvidea included kernel image and modules with their package because if I look at the ‘Source’ folder in their package it only includes dts and dtsi files. This leads me to believe that for their carrier board I only need the dtbs built from these device tree source files. If my assumption is correct and now that I have a kernel image and modules manually built that includes edits for D435 camera then may be if I just ‘copy’ these to TX2, it should work with both what do you think?

In short Auvidea support is embedded in dtbs while camera support is embedded in additional kernel modules and kernel image. I have a system running with Auvidea support already, so if I can only replace the manually built kernel and modules for camera in the running system it might work. Then again it does sound a very long shot.

Sorry, this is a lot longer than needed, but it is likely other people will also need to understand what is happening (which goes beyond a simple recipe for using Auvidea’s carrier)…

Changes to .c/.h files or .dts/.dtsi files demonstrate that the Intel features are added only through kernel source, and not device tree. Any device tree changes are apparently only for the Auvidea carrier board.

This shows that the Auvidea features are added only with the device tree, and do not change kernel source (kernel “source” is a subset of the kernel “package”…device tree is the complementary subset of the kernel “package”).

Both Intel and Auvidea changes are packaged within the kernel package, but are essentially autonomous from each other so far as build dependencies go. Neither build cares that the other exists or does not exist (but there will be a run time failure if they use incompatible configurations…configuration is what these share in common).

I can only speculate that the reason for Auvidea providing kernel Image and modules (even if only device tree changed) is due to a change in “uname -r”, which in turn would change where modules are searched for. Can you confirm what the “uname -r” is with the Auvidea changes in place?

You could boot an unmodified Jetson on a dev carrier, save a copy of “/proc/config.gz”, and then save the “/proc/config.gz” from booting an Auvidea kernel on the Auvidea carrer…then compare the two configs. It would be a really good idea for you to save a copy of both of those configs for later reference anyway…you may need to know at some later date when you build a kernel. While you are at it, save the extracted device tree while booted to each kernel. Example:

dtc -I fs -O dts -o extracted_`uname -r`.dts /proc/device-tree

(you will probably want a copy of these to examine in the future when you no longer have the original kernel and device tree…every developer should save a copy of this in a safe place from the original unmodified kernel and from any modification you are keeping)

As mentioned, the kernel source contains both the kernel itself and the device tree, but both being present is just for packaging convenience since drivers and device tree are related…but they are independent in actual build and often edited independently. If only device tree changed, then it is reasonable that Auvidea’s source is just device tree files. If you were to build your own kernel, e.g., “make Image” (zImage is the same thing, it is just compressed…not used with U-Boot on the 64-bit releases of Jetsons), then the device tree files could be completely missing and the build would never know (nor would the compiler care…it is only when the “dtc” script is used that device tree files matter). Conversely, if config is set up correctly and you are just building device tree files, then all of the kernel source files could be completely missing and dtc would not care.

To integrate all of the changes it seems that basically, if you take the existing TX1TX2 kernel source from source_sync.sh, apply Intel’s files and patches to the C portions, and then overlay Auvidea’s device tree files on top of that, then you would have everything integrated into a single package. Should this source tree be first correctly configured, then the build targets for Image, modules, and dtbs (“dtbs” is a build target, e.g., “make dtbs”; “Image” is a target, e.g., “make Image”; “modules” is a target, e.g., “make modules”), then your result would be everything in one package. Placing the new Image in the right place, and placing the new device tree files in the right place, followed by flashing, should result in everything going where it is needed (“flash.sh” copies the Image and device tree files based on arguments it is given…the rest comes from “rootfs/”).

There may be other adjustments to account for, e.g., if the new Image has a different “uname -r”, then the “rootfs/lib/modules/$(uname -r)/” would need modules copied there as well (“flash.sh” does not know the “uname -r” of a modified kernel, it only knows where the script originally says to place it…“rootfs/lib/modules/4.4.38-tegra/”). The caution to point out is that at the time of flash building the system.img that the actual “rootfs/boot/Image” and other boot related files are edited, so you can’t just put the Image there…it’ll be in one of the “Linux_for_Tegra/kernel/” subdirectories, and options to flash.sh will determine which device tree files and Image file are copied in to “rootfs/” (you edit what is in “Linux_for_Tegra/bootloader/”, “Linux_for_Tegra/bootloader/”, or other locations of “Linux_for_Tegra/” which are not “rootfs/” if you want to flash a modified boot configuration…all other changes go into “rootfs/”). I can’t tell you exactly what would change, I’ve never used Auvidea’s hardware.

For reference, if you are using L4T R28.2 (see “head -n 1 /etc/nv_tegra_release”), then the R28.2 source_sync.sh can obtain the full kernel package via:

./source_sync.sh -k tegra-l4t-r28.2

If you cd in to the “sources/kernel/kernel-4.4/”, you will see that this is from a git repository. “git status” can look at changes. Should you unpack the Intel patches and Auvidea device tree, then in theory it will point out exactly what has changed. Only the changes which the specific build configuration uses (which source_sync.sh won’t have…this is why you copy original “/proc/config.gz” and note “uname -r” for CONFIG_LOCALVERSION) will go into your TX2. If you log the flash of the TX2, then you will know which files were copied into the system.img.

Thanks linuxdev. Your detailed response is much appreciated. I checked the uname -r on the TX2 and it is 4.4.38-tegra. So Auvidea didn’t modify the local version. I would like to point out that though I have flashed the TX2 with package from Auvidea (Kernel, modules, dtbs) it is still running on my devkit. Its just convenient for me to reflash multiple times. Does that make any difference to .config while running?

I am now going to go ahead with combining dtbs from Auvidea and kernel image with patches from Intel. One last thing before I do that. Let’s assume for a moment that there were some additional modules added to kernel image by Auvidea. Wouldn’t it be possible to extract the .config from running system and then edit it further to add modules for Intel. This will then have all the additional modules required by both Auvidea and Intel and once I build an image using this .config it will be complete. Just a thought, I can be way off :)

The device tree might have some components in common, but I’d be surprised if everything works if running a TX2 on a dev kit after Auvidea device tree is added. For example, USB and GPIO might fail or various other I/O might fail. Some components of device tree are specific to the carrier board it is on…you cannot mix this without unexpected consequences. Flashing it from the dev kit and moving it to the Auvidea board is ok, but actual run using the wrong tree on the wrong carrier is problematic.

The kernel and its config is possibly (“probably”) portable across the two carriers and ok. Certainly the Intel RealSense kernel mods would be ok on either carrier if the mods are ok on any carrier.

If modules are added (not integrated kernel features), then those modules would be ok somewhere in “/lib/modules/$(uname -r)/”. The Image file itself would not be changed. If you change a feature which is integrated (and not as a module), then the Image file itself must be changed.

Extracting the running system’s config and modifying with additional Intel module additions is a very good idea. Just be careful to use a configuration editor (such as “make nconfig”), and don’t edit the .config file directly since there may be dependencies…and those dependencies change depending on kernel version, so you can’t rely on direct edits without a dependency-aware editor. Adding modules is low risk…it is when you modify by removing integrated non-module features that the Image itself will need to be replaced, and possibly all of the modules along with that change.

Hi linuxdev,

I read somewhere in the forum here few users reported that dev kit works with Auvidea firmware. As you mentioned there could be some issues with IO but I haven’t tried anything so far apart from USB.

Alright, as planned, I extracted the .config from running system with kernel provided by Auvidea and edited it using nconfig to add modules for Intel RealSense. Next I patched the kernel source for RealSense. After patching I built the kernel and modules. Started Jetpack 3.2.1 and before flashing on Target I replaced the manually built kernel and modules plus dtb files from Auvidea in the respective Linux_for_Tegra subfolders. Hit continue on the JetPack and voila! it flashed fine and then installed samples and libs for CUDA, Visionworks etc.

So hopefully I have a system now that has kernel modifications and modules from both Intel and Auvidea plus device tree from Auvidea.

I will be going to mount the TX2 SOC on Auvidea board and attach the RealSense camera to it next, probably tomorrow and report back if it actually works :)

One question, when .config is extracted from the running system and a kernel is built using it, does the build process compiles both the integrated kernel features and external modules or is it just the modules?

Thanks

Other than CONFIG_LOCALVERSION this is an exact match for how the running kernel was configured when it was built. This includes modular and integrated features. Thus “make modules” would result in the same modules being built. “make Image” would make an exact duplicate “Image” if CONFIG_LOCALVERSION is set to match from the same source tree.

The “source_sync.sh” version may actually append a pesky “+” on the end of the “uname -r” in addition to the CONFIG_LOCALVERSION, but otherwise even this would be an exact match on which integrated and modular features are built.

Hi,

I am happy to report that I have successfully mounted the Tx2 on J120 and attached D435 camera to it. Camera is identified as USB 3.0 device on J120. This confirms that kernel properly supports the board. One of the common issues users faced was that OS couldn’t recognize the USB 3.0 on one of J120 ports. I also verified that FAN was being properly controlled by the OS, this was the other common issue.

Camera specific patches also seem to work as I can see that uvc kernel module is able to support different video formats that were not part of the stock uvc module. So all in all I think it is a success.

I would like to thank @linuxdev for helping me along the way. I was stuck at configuring the kernel modules since I was directly editing .config and kernel build was failing because manual edits were not aware of any missing dependencies. Following the advice to use nconfig was the main missing link. I am a happy customer now. Once again many thanks to @linuxdev for helping and resolving my doubts every step of the way. you rock!! :)

I am also looking to use a D435 on a TX2/J120. Given that the device tree and the kernel sources are changed independently (device tree for J120 patches, kernel sources for realsense patches), could the patched kernel be built with the Intel patches, but without the Auvidea patches, and installed on an already configured TX2/J120? I.e. Build the kernel as described in JetsonHack’s on device build method (GitHub - jetsonhacks/buildJetsonTX2Kernel: Build the NVIDIA Jetson TX2 Kernel on the device itself) and copy the image, rather than flashing.

To be clear, you are unlikely to get an Auvidea board working without Auvidea’s device tree changes. It would be highly advisable to at first work only on getting the Auvidea patches in place before working on additional peripheral patches. I don’t know about the Intel patches, and do not have an Auvidea board, so there isn’t a lot more I could tell you about it.

My apologies, let me clarify. I have the TX2/J120 working with the Auvidea provided firmware.

It was my understanding from the above posts that since the RealSense patches were strictly for the kernel source, and the Auvidea patches were strictly for the device tree, that the make targets were somewhat independent.

My ultimate question is whether or not I could patch the L4T kernel with the RealSense patches, build that image, and install it without affecting the working J120 device tree, which is in a separate partition iirc.

Best,

Hal

Your understanding is correct, but there is a “catch”. Drivers may rely on the device tree being set up as expected. Some boards might use the same driver on the same hardware which is arranged at a different offset in memory, or a different GPIO lane dedicated to the device. If the hardware is present where the driver expects, then it should work. The driver can be told where to expect things via the device tree. Stages prior to the driver running can also perform the actual hardware initialization steps (telling the driver what the setup is without having performed the setup won’t help), later stages use that initialization.

So yes, assuming Auvidea is purely device tree edits (and all I’ve seen says this is the case…but I have no actual example to work with), then your RealSense patches should be independent of the parts which are purely for Auvidea.

Just be sure that if you build a new kernel that the kernel matches the currently running kernel. The same kernel source code with two different configurations can be a complete mismatch.

I would just like to point out that I did not assume that Auvidea’s changes were limited to device tree only. Therefore, I first flashed jetson with Auvidea’s kernel, got the config file from the flashed system to get the changes that Auvidea might have made to the kernel prior to building it and then used this config file as basis to add edits for RealSense in addition to patching the kernel sources for RealSense.

Gotcha, I will try and use that as the existing config as the basis as well.

I will try it out and report back - made a full device clone first, in case I brick it.

Thank you both so much for your help!

I was unable to make the compile-in-place method work, but I was able to follow nouman.hanif/linuxdev’s method of flash auvidea firmware → extract config.gz → patch kernel and adjust config, and flash. The newly built kernel seems to work, but I am still unable to get it to register as USB 3.0, falls back to 2.0. @nouman.hanif was there any other USB3 related patching steps that you took on? It seems to be a chronic issue with these boards. Fan seems to work though, leading me to believe that the Auvidea device tree is correct and intact.

When I rebuilt the kernel, I only copied over the Image, and did not build dtsb, leaving the auvidea device tree in place - could this be the issue?