SeeedStudio j4012 (Orin NX 16GB) & JP6 ^ Stereolabs capture card

I’m on a SeeedStudio j4012 (Orin NX 16GB). It came with Jetpack 5.1.1, and has worked with the driver for capture card Stereolabs link DUO.

Now I have on it Jetpack 6.0 L4T 36.3.0 and the driver 1.1.0 for DUO capture card.
I get this:

virtual@viojetson:~$ sudo dmesg | grep zedx
[sudo] password for virtual:
Sorry, try again.
[sudo] password for virtual:
[   12.902332] sl_zedx: disagrees about version of symbol tegracam_v4l2subdev_unregister
[   12.902343] sl_zedx: Unknown symbol tegracam_v4l2subdev_unregister (err -22)
[   12.902391] sl_zedx: disagrees about version of symbol tegracam_v4l2subdev_register
[   12.902393] sl_zedx: Unknown symbol tegracam_v4l2subdev_register (err -22)
[   12.902406] sl_zedx: disagrees about version of symbol tegracam_device_unregister
[   12.902407] sl_zedx: Unknown symbol tegracam_device_unregister (err -22)
[   12.902446] sl_zedx: disagrees about version of symbol tegracam_get_privdata
[   12.902447] sl_zedx: Unknown symbol tegracam_get_privdata (err -22)
[   12.902467] sl_zedx: disagrees about version of symbol camera_common_mclk_enable
[   12.902468] sl_zedx: Unknown symbol camera_common_mclk_enable (err -22)
[   12.902479] sl_zedx: disagrees about version of symbol tegracam_set_privdata
[   12.902480] sl_zedx: Unknown symbol tegracam_set_privdata (err -22)
[   12.902489] sl_zedx: disagrees about version of symbol tegracam_device_register
[   12.902490] sl_zedx: Unknown symbol tegracam_device_register (err -22)
[   12.945327] sl_zedxone_uhd: disagrees about version of symbol tegracam_v4l2subdev_unregister
[   12.945339] sl_zedxone_uhd: Unknown symbol tegracam_v4l2subdev_unregister (err -22)
[   12.945390] sl_zedxone_uhd: disagrees about version of symbol camera_common_g_ctrl
[   12.945391] sl_zedxone_uhd: Unknown symbol camera_common_g_ctrl (err -22)
[   12.945411] sl_zedxone_uhd: disagrees about version of symbol tegracam_v4l2subdev_register
[   12.945413] sl_zedxone_uhd: Unknown symbol tegracam_v4l2subdev_register (err -22)
[   12.945425] sl_zedxone_uhd: disagrees about version of symbol tegracam_device_unregister
[   12.945426] sl_zedxone_uhd: Unknown symbol tegracam_device_unregister (err -22)
[   12.945462] sl_zedxone_uhd: disagrees about version of symbol tegracam_get_privdata
[   12.945463] sl_zedxone_uhd: Unknown symbol tegracam_get_privdata (err -22)
[   12.945479] sl_zedxone_uhd: disagrees about version of symbol camera_common_mclk_enable
[   12.945480] sl_zedxone_uhd: Unknown symbol camera_common_mclk_enable (err -22)
[   12.945492] sl_zedxone_uhd: disagrees about version of symbol camera_common_mclk_disable
[   12.945493] sl_zedxone_uhd: Unknown symbol camera_common_mclk_disable (err -22)
[   12.945501] sl_zedxone_uhd: disagrees about version of symbol tegracam_set_privdata
[   12.945502] sl_zedxone_uhd: Unknown symbol tegracam_set_privdata (err -22)
[   12.945511] sl_zedxone_uhd: disagrees about version of symbol tegracam_device_register
[   12.945512] sl_zedxone_uhd: Unknown symbol tegracam_device_register (err -22)

Some might say that my module deals with video streams different from Oficail Nvidia boards since it has HDMI output instead of Display port.

If you have any hints, please share!

Is sl_zedx the driver, or is it the application itself? I’m the wrong guy to ask about the particular hardware, but an unknown symbol means it needs to link to something else which is not there.

Probably that is the driver (I don’t know), and if it is, then it means your kernel needs another kernel module or other format of driver which has the function it is looking for (such as tegracam_v4l2subdev_register; this is why I think it is probably a kernel driver and not so likely that it is a user space library).

If this is in fact a user space program, then the concept is the same, but the missing symbol would be provided by adding a library and not by adding a kernel module/driver. Someone from NVIDIA could answer if this is a missing library or a missing kernel module/driver.

You could find out ahead of time though what libraries your user space program wants. If the program is named “app” and is located at “/usr/bin/”, then this would tell you what user space libraries it wants:
ldd /usr/bin/app

You can get a list of user space libraries which are within the default linker path via:
ldconfig -p

The library name from ldd should show up in ldconfig -p. If all ldd named files exist somewhere in ldconfig -p, then your user space is complete and only the kernel driver/module remains.

1 Like

Thank you for sharing the info.

I think the ‘sl_zedx’ is the drive of the GMSL2 capture card.

Bellow is the error log of the diagnose app of the camera.

dmesg_2.log (15.3 KB)

I will emphasize I can’t give a complete answer. The following file or information will be useful:

  • cat /etc/nv_tegra_release (just a copy and paste)
  • cat /etc/nv_boot_control.conf (just a copy and paste)
  • dmesg 2>&1 | tee dmesg.txt (then attach file dmesg.txt to the forum).
  • Attach a copy of “/etc/systemd/system/zed_x_daemon.preload”.

From the attached file I have some observations:

  • This device uses i2c control. This is similar to an ordinary serial UART, but it has to have addressing and setup. The code to deal with this is a kernel driver. That kernel driver appears to be either missing or not configured. The files in “/dev” of the name format “/dev/i2c-<number>” are created by the driver itself when the driver is loaded.

  • Often the i2c setup and knowledge from the driver of the requirement to load is from the device tree. Another related software you might hear which is related to this is the PINMUX spreadsheet, which just has macros for customizing GPIO (and the i2c is one possible GPIO function). This tends to not be portable across different carrier boards, nor for example across different hardware like the RPi and Jetson. An Orin NX on a dev kit carrier board likely differs from a commercial module on a third party manufacturer’s carrier board. It is very important to know the exact model of carrier board you have; most will require i2c device tree information from the manufacturer of that carrier board. If everything here is purely NVIDIA-produced, then NVIDIA can answer about the i2c setup.

  • That log shows:
    Process "insmod /usr/lib/modules/5.15.136-tegra/kernel/drivers/stereolabs/zedx/sl_zedx.ko" outputs "insmod: ERROR: could not insert module /usr/lib/modules/5.15.136-tegra/kernel/drivers/stereolabs/zedx/sl_zedx.ko: Invalid parameters\n"
    There are two related concepts here. The first is that if the module is not compiled to work with your specific kernel, then it likely cannot load. The kernel configuration at the time of compiling a module needs to match the kernel configuration of the kernel that this will load into. That isn’t the listed error, but it might just be an inadequate message. The part about “Invalid parameters” is that often modules are provided parameters to pass to the driver as the driver loads. Sometimes those are provided in the load command itself, but sometimes those are provided by the device tree. This could also be a device tree issue, or it could be about being compiled against the incorrect kernel. I’m betting it is about arguments, but we don’t have the “dmesg” log.

  • The lack of i2c setup probably would not stop the kernel module from loading, although it might provide an argument to the kernel module (not likely; I’m betting on device tree).

What is the exact module and carrier board? Does this module have eMMC? Is there an SD card slot on the module itself? Or is any SD card slot on the carrier board? This has to be identified before the files and information I’m suggesting you attach will be fully useful.

1 Like

Thank you again @linuxdev

I’ll break the answer to your questions in two parts if I may.
The reason is that I have to reflash the device with JP6 again, I put again JP5.1.1 to be 200% sure that the capture card really works.

Carrirer board is : J401 Carrier Board with Orin NX16Gb module from Nvidia.

It has NVme, no SDCard or eMMC.

On the manufacturer github there are some configurations files…
For the moment I try to reflash JP6 with the github method, first time I have used the wiki page from the manufacturer.

This is the CONF file for example

The carrier board is:

Most of the specs are here:


AI Performance

reComputer J4012 - Orin NX 16GB, 100 TOPS

reComputer J4011- Orin NX 8GB – 70 TOPS

GPU

1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores

CPU

8-core Arm® Cortex®-A78AE v8.2 64-bit CPU (Orin NX 16GB)

6-core Arm® Cortex®-A78AE (Orin NX 8GB)

Memory

16GB 128-bit LPDDR5 (Orin NX 16GB)

8GB 128-bit LPDDR5 (Orin NX 8GB)

Video Encode

1x 8K30 (H.265) | 2x 4K60 (H.265) | 4x 4K30 (H.265) | 9x 1080p60 (H.265) | 18x 1080p30 (H.265)

Video Decode

1x 4K60 (H.265) | 3x 4K30 (H.265) | 6x 1080p60 (H.265) | 12x 1080p30 (H.265)

DL Accelerator

2x NVDLA v2

Carrier Board

Storage

1x M.2 Key M PCIe (M.2 NVMe 2280 SSD 128G included)

Networking

Ethernet

1x RJ-45 Gigabit Ethernet (10/100/1000M)

M.2 KEY E

1x M.2 Key E for WiFi/Bluetooth module

I/O

USB

4x USB 3.2 Type-A (10Gbps)

1x USB2.0 Type-C (Device Mode)

Camera

2x CSI (2-lane 15pin)

Display

1x HDMI 2.1

Fan

1x 4 pin Fan Connector (5V PWM)

CAN

1x CAN

Multifunctional Port

1x 40-Pin Expansion header

1x 12-Pin Control and UART header

RTC

1x RTC 2-pin, supports CR1220 (not included)

Power

9-19V DC

Mechanical

Dimensions (W x D x H)

130mm x 120mm x 58.5mm (with case)

Installation

Desk, wall-mounting

Operating Temperature

-10℃~60℃

Warranty

1 Year

There are the rest of your requests:

vivio@viojet:~$ cat /etc/nv_tegra_release
# R36 (release), REVISION: 3.0, GCID: 36191598, BOARD: generic, EABI: aarch64, DATE: Mon May  6 17:34:21 UTC 2024
# KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia



vivio@viojet:~$ cat /etc/nv_boot_control.conf
TNSPEC 3767-300-0000-M.1-1-1-recomputer-orin-j401-
COMPATIBLE_SPEC 3767-000-0000--1--recomputer-orin-j401-
TEGRA_BOOT_STORAGE nvme0n1
TEGRA_CHIPID 0x23
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

This file does not exists “/etc/systemd/system/zed_x_daemon.preload”

dmesg.txt (71.3 KB)

dmesg.txt
``` is now attached

I’m just going through this and commenting as I see it; this might not be complete at the start and something might change as I get further in.

About this:

If you have a Jetson Orin NX, it means it has a Jetson module, and that module is mounted on a separate carrier board. There is no such thing as a Jetson module which does not have either (A) an SD card mounted on the bottom of the module (which you might not use, and might not see), or (B) eMMC. One of those two must be true, without exception. Commercial modules sold with third party carrier boards will always use eMMC, so I think the eMMC exists, but you don’t see it because you are using external devices. The eMMC might not be used, but as an experiment, what do you see from this (eMMC models tend to have many partitions which contain the equivalent of the BIOS, but in software form instead of having an actual hardware BIOS):
lsblk -f

Knowing the model is mandatory before knowing about flashing. Incidentally, the term used in the description of the board is that it is “with NVIDIA® Jetson Orin™ NX SoM”. The “SoM” part says it is a System on a Module. It is the module format that must have either SD card slot or eMMC. You could break this “either SD or eMMC” if and only if you are directly using the Tegra chip (and NVIDIA probably requires purchasing around $1 million USA dollars of the chip before it would sell individual chips; reComputer is not using the chips directly, it is using modules).

Any time you use an external media you are likely to require an initrd flash instead of just a straightforward flash. This would be in a tools subdirectory of the host PC’s “Linux_for_Tegra/” flash software. However, you won’t use this flash software.

Since this is a reComputer, you must use their software, or their flash instructions. I believe this is not an exact match to the dev kit carrier board, so at minimum there will be a patch to the NVIDIA flash software. Seeed’s reComputer instructions and any downloads should take precedence over the NVIDIA flash software. Not using the Seeed Studio software would likely result in parts of the Jetson being non-functional.

Tip: The file named “recomputer-orin-j401.conf” might create a flash target of “recomputer-orin-j401”. Flash targets all have a .conf file, and in command line, one names the name of the file, but without the .conf. Do check what the Seeed Studio instructions are though.

Getting to the actual dmesg log it really looks like the drivers were compiled against a different kernel:

[   11.296974] sl_zedx: disagrees about version of symbol tegracam_v4l2subdev_unregister
[   11.296984] sl_zedx: Unknown symbol tegracam_v4l2subdev_unregister (err -22)

How did you add the zed drivers? Those used in L4T R35.x will differ from those used in L4T R36.x. If you are using a binary driver from Stereolabs.com, then it has to match the kernel being used. L4T R36.x has a newer kernel than R35.x.

1 Like

Once again thank you for looking the solution for this!

Regarding eMMC, see here the result

vivio@viojet:~$ lsblk -f
NAME FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0
     squash 4.0                                                    0   100% /snap/bare/5
loop1
     squash 4.0                                                    0   100% /snap/chromium/2906
loop2
     squash 4.0                                                    0   100% /snap/core22/1614
loop3
     squash 4.0                                                    0   100% /snap/core24/493
loop4
     squash 4.0                                                    0   100% /snap/cups/1059
loop5
     squash 4.0                                                    0   100% /snap/gnome-46-2404/49
loop6
     squash 4.0                                                    0   100% /snap/gtk-common-themes/1535
loop7
     squash 4.0                                                    0   100% /snap/mesa-2404/142
loop8
     squash 4.0                                                    0   100% /snap/snapd/21761
loop9
     vfat   FAT16 L4T-README
                        1234-ABCD                                           
zram0
                                                                            [SWAP]
zram1
                                                                            [SWAP]
zram2
                                                                            [SWAP]
zram3
                                                                            [SWAP]
zram4
                                                                            [SWAP]
zram5
                                                                            [SWAP]
zram6
                                                                            [SWAP]
zram7
                                                                            [SWAP]
nvme0n1
│                                                                           
├─nvme0n1p1
│    ext4   1.0         76c23f85-be8b-492c-98ba-2513a5f8aba7  100,5G     8% /
├─nvme0n1p2
│                                                                           
├─nvme0n1p3
│                                                                           
├─nvme0n1p4
│                                                                           
├─nvme0n1p5
│                                                                           
├─nvme0n1p6
│                                                                           
├─nvme0n1p7
│                                                                           
├─nvme0n1p8
│                                                                           
├─nvme0n1p9
│                                                                           
├─nvme0n1p10
│    vfat   FAT32       56EB-1B73                                           
├─nvme0n1p11
│                                                                           
├─nvme0n1p12
│                                                                           
├─nvme0n1p13
│                                                                           
├─nvme0n1p14
│                                                                           
└─nvme0n1p15

The version of the driver / kernel. For me it seems to be identical

vivio@viojet:/usr/lib/modules/5.15.136-tegra/kernel/drivers/stereolabs/max96712$ modinfo sl_max96712 
filename:       /lib/modules/5.15.136-tegra/kernel/drivers/stereolabs/max96712/sl_max96712.ko
version:        ..
license:        GPL v2
author:         STEREOLABS <support@stereolabs.com>
description:    IO Expander driver max96712
srcversion:     455104127E0F6C19395EF6F
alias:          i2c:sl_max96712
depends:        
name:           sl_max96712
vermagic:       5.15.136-tegra SMP preempt mod_unload modversions aarch64
vivio@viojet:/usr/lib/modules/5.15.136-tegra/kernel/drivers/stereolabs/max96712$ uname -r
5.15.136-tegra

I’m thinking about this fact mentioned on nvidia forum, but I’m not sure exactly how to implement it.

I don’t see the eMMC in your lsblk, but I missed a detail in the command. It needs the “-a” option to say “all” even if not used. What is the output of “lsblk -a -f”? I think an SD card controller won’t show up at all without the SD card. Does anyone know of an Orin NX module with neither SD card slot on module, nor eMMC? I don’t think such a thing exists. For the moment I will treat it only as an eMMC model of module because third party carrier boards never use the “SD card on module” format.

Keep in mind that the reason for needing the exact type of module and carrier board is that instructions and software change with each type. We know it is SeeedStudio, and I assume an eMMC model despite being flashed to use NVMe.

From the modinfo, in combination with the previous note of i2c errors, I’m thinking that if i2c is solved, then the module might work. There is also reference to symbol symbol tegracam_v4l2subdev_*. A summary of issues that block this from working:

  • The Linux side complains about the tegracam_v4l2subdev_.
  • The user space program complains about lack of i2c.

The i2c issue can be described as this: If the i2c controller on the Jetson does not exist, then this would likely be due to the device tree being wrong. Using NVIDIA’s device tree from flash software is high on the list of likely causes; the i2c depends on the carrier board, which means this might be something needed from Seeed Studios. This could also be because the device itself (which has the other end of i2c hardware, it is a host/client relationship) is not responding. With third party carrier boards I’d give odds of about 99% that it is a device tree issue from using the NVIDIA device tree, but needing the Seeed Studio device tree. But perhaps 1% odds that the lack of the kernel module causing the device’s i2c to not respond.

Where did you get your device tree? Was your JetPack/SDK Manager software patched first with the Seeed Studio software? Does the Seeed Studio information on this exact carrier board have its own downloads, or does it say to use the NVIDIA flash software directly?

Incidentally, that URL you mention just tells how to take a new kernel and module build on the host PC (cross compiled) and install it to the flash software’s basis for the filesystem. It installs the new kernel and modules for future flashes. Configuration would likely still be required. You certainly don’t need to flash the system to install modules, but if you’ve used NVIDIA’s flash software and it is really Seeed Studio’s software you need, then you must reflash again anyway.

The tests conducted not are by installing the jetpack6 with this method:

But I’ve made a breakthrough:
The first success was when i use the shell commands for Stereolabs for their ZED BOX Orin device (it has an ORIN NX 16GB module with HDMI output).

You can find it here:

https://stereolabs.sfo2.cdn.digitaloceanspaces.com/utils/zed_boxes/zedbox_onx_usb_flash_6.0.sh

I’ve tried to see what commands they have, but for now I think those are the important ones:

1. Download and prepare the Linux_for_Tegra source code (untar the files)

wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v3.0/release/jetson_linux_r36.3.0_aarch64.tbz2

tar xf jetson_linux_r36.3.0_aarch64.tbz2

2. Download and prepare sample root file system (untar the files with SUDO in “Linux_for_Tegra/rootfs/”

wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v3.0/release/tegra_linux_sample-root-filesystem_r36.3.0_aarch64.tbz2

sudo tar xpf tegra_linux_sample-root-filesystem_r36.3.0_aarch64.tbz2 -C Linux_for_Tegra/rootfs/

3. GEt & Unpack ZED Box device tree binaries and kernel modules

wget https://stereolabs.sfo2.cdn.digitaloceanspaces.com/utils/zed_boxes/zedbox_device_trees_363.tar

tar -xf zedbox_device_trees_363.tar

4. ST

Get & unpack ZED Box device to “Linux_for_Tegra/rootfs/”

wget https://stereolabs.sfo2.cdn.digitaloceanspaces.com/utils/zed_boxes/zedbox_rootfs_363.tar

sudo tar -xf zedbox_rootfs_363.tar

5. ST

cd Linux_for_Tegra/

sudo ./apply_binaries.sh

6.

echo “Setting hostname:”

– the command is not clear for me: sudo echo ${GENERIC_ZBOX_NAME} | sudo tee ./rootfs/etc/hostname

7. Applying the prerequisites

sudo ./tools/l4t_flash_prerequisites.sh

8. Creating the default user

sudo ./tools/l4t_create_default_user.sh -u user -p admin -n user-ubuntu -a --accept-license

9. Starting the flash process

*sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 *

*-c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml " *

–showlogs --network usb0 p3509-a02-p3767-0000 internal

There is a difference from SEEED github:

Can you spot the differences between SEEED and Stereolabs?

At the moment the device can capture, but is not quite perfect and since is not tailored for its carrier module for sure there are more things mixed up.

Just some notes as I read your post…

  • I see the zedbox_onx_usb_flash_6.0.sh script adds any patch to some starting version of NVIDIA software and performs an initrd flash. I assume that because this is from StereoLabs that they intend to use this with either a Jetson dev kit, or some specific board. I did not look far enough to know that answer. If you know what the StereoLabs script is intended to be used on (which probably is not your Seeed carrier board) it would probably be useful to know. Unless the intended target is your particular carrier board and module combination, then the wrong content would be flashed (their Zed patches would be correct, but applied to the wrong rootfs).
  • For the first GitHub URL you show, are you saying you did use that? Or that you did not do that? Seeed devices would use this software, which is an edit of the NVIDIA flash software. This is what the Zed patches should be applied to, except that if there are any device tree edits, then those edits may need modification for the different device tree Seeed uses compared to a dev kit. I’m kind of guessing because I don’t know intended hardware combinations for those URLs.
  • Most every manufacturer will start with the NVIDIA root filesystem, and then edit. However, device trees can be in more than one location, and can be read from different locations. Sometimes a full serial console boot log is useful to know which tree is actually being read. If Zed does not edit device trees, then the boot content (which is mostly outside of the rootfs) can be purely that of Seeed’s software. If Zed does require some different firmware setup (which is in the device tree), then you’d have to have a device tree which is compliant with both Seeed’s device tree and Zed’s. The device tree helps to find non-plug-n-play hardware, and to pair a driver with it. When the device tree is not correct, then parts of the system will not function, e.g., maybe GPIOs. If Zed requires some special hardware setup, e.g., GPIO, then that has to be edited; since I think Zed is USB (plug-n-play), it is likely Zed does not edit (nor need) changes tot he device tree. Some other camera formats (not USB, but perhaps CSI) would require device tree edits.
  • You should probably start with the Seeed flash and install to get your carrier board and module combination to use the correct firmware.
  • Whatever kernel is installed, if you choose to compile a module yourself, then it must be compiled against that running kernel’s configuration. Perhaps Seeed has a different configuration than the NVIDIA dev kit kernel, and if Zed assumes the NVIDIA kernel, but it is really modified for Seeed (and I don’t know if it is modified or not), then you’d have a module failure. If the Seeed and NVIDIA kernels are the same configuration, then such a compile would “just work” (because they’d be the same configuration, compiling against either is compiling the same thing).

This can get rather complicated to experiment, but you might want to record the steps of flashing once with Seeed software, and then once with Zed software (then compare the logs). If you append any command line with " 2>&1 | tee log_flash.txt", then you will record the flash output. As an example:
sudo ./zedbox_onx_usb_flash_6.0.sh 2>&1 | tee log_zed_flash.txt

You could do similar with a flash of Seeed software, but change the file name to “log_seeed_flash.txt”, and attach the log files to the forum thread.

Have you asked Zed about whether their script, and whether it would care that you are applying it to a Seeed board?

Incidentally, you can save the kernel module compile until the end. You only need that module when you actually run the software the module is for, and that software won’t be used in boot, nor in regular function of the Jetson. Camera plug-in is when that module is needed. If everything is flashed and set up correctly, including device tree, you can then worry about the module (which would actually be simple).

HI @linuxdev

Thanks for the rescue :)

Stereolabs tries to stick to Nvidia kits and some limited partner list. I understand this approach, since everybody has limited resources.

They at first did not believe that someone has managed to make the SEEED work on their camera with JP6. I’ve explained the method and maybe I’ll get some limited help from them.

I have done some things that can be useful:

  1. I managed to install JP6 on J4012 by using the Nvidia SDK :)
  • downside: CUDA did not install properly, no WIFI
  • upside: after installing the ZED drivers the camera has WORKED :)
  • so, it seems that on pure JP6 the things works
  1. I listed to your advice and I’ve installed on J4012 the STEREOLABS modified JP6.
  • here I have WIFI (they inject some intel drivers and I have a AX200 WIFI board)
  • camera works

You can have the normal log
flash_3-1_ZED_BOX_method.log (50.1 KB)

And the one with commands, but they are not that clear for me
log_zed_flash.txt (3.9 MB)

Regarding SEEED you want me to follow their GIthub page in order to install JP6?

You can skip to the “Summary:” at the end, but what is between will help you to understand how to compare a Seeed device tree to the ZED/Stereo Labs device tree. Mostly it looks like ZED/Stereo Labs has used overlays, which is quite useful. However, the two trees which I think are the same need to be compared first.


If you can extract just the device tree from the Seeed software, and install that to the NVIDIA install so that it uses this instead of NVIDIA’s, then much non-functioning parts of the carrier board are likely to start working. The NVIDIA device tree will be exactly the same as the Seeed one for much of the carrier board; Seeed would not have changed all parts of the carrier board, but would have instead changed only the parts that helped them route PCB lanes. This is the vast majority of what the Seeed flash software would have modified. Perhaps a different kernel, but this too would start with a kernel configuration matching NVIDIA’s, and then editing only minor configuration changes. This may not fix “everything”, but it could, and it would go a long ways towards making other parts functional.

If you remove the “quiet” anywhere in “/boot/extlinux/extlinux.conf”, then you might be able to save a serial console boot log and look up any “dtb” words in the log to get an idea of what device tree is loading. However, if this loads from some other device as binary data, then you might not see “dtb” as a key word.

The log file I talk about next is your first one, flash_3-1_ZED_BOX_method.log (50.1 KB)

There is the reverse of this as well (which is what you indicated you have done): You have installed the Stereo Labs modified JP6, which has all kernel changes (if any; there might not be any), including device tree, and then “retrofit” the ZED/Stereo Labs content onto this. Details below relate to your logs under this scenario:

  • I see a tegraflash.py command. Within this it mentions:
    • --bldtb tegra234-p3768-0000+p3767-0000-nv.dtb
    • kernel_dtb tegra234-p3768-0000+p3767-0000-nv.dtb
    • I don’t think this is relevant: pva_fw nvpva_020_sigheader.fw.encrypt
      The reason is that I think this is just the “signing” stage of content rather than naming actual content.

The above is logged as part of the initrd. This initrd serves as a kind of adapter to allow booting to the NVMe or other external media. Those device trees would be known by both the Linux kernel and boot stages. However, this implies bootloader stage “--bldtb tegra234-p3768-0000+p3767-0000-nv.dtb” (that abbreviation bldtb refers to bootloader). You’re not really concerned with what the bootloader tree does, although any environment setup during boot will be inherited by the Linux kernel. The part of the tree you are interested in is the one which the Linux kernel uses directly. Thus I think that this file is relevant:
tegra234-p3768-0000+p3767-0000-nv.dtb

I don’t know if this is different between what the Stereo Labs would have flashed (in fact it seems odd that Stereo Labs would provide actual flash software; please correct me if I’m wrong about that). Also, can I get verification what the wiring is for the ZED camera? Is it USB? Is it CSI? If it is USB life is much simpler since the camera itself won’t depend on a device tree. A USB camera could still indirectly depend on the device tree because it could require use of hardware that is lost if the device is wrong.

Getting back to tegra234-p3768-0000+p3767-0000-nv.dtb, this tree might or might not have the same name when flashing with NVIDA JP and when flashing with Seeed software. I don’t know. If it is the same name, then you need to find out if they are actually the same thing, or if instead they are edits with the same file name. I can provide ways of doing this if you find two versions of tegra234-p3768-0000+p3767-0000-nv.dtb. If they use different (but similar) device tree file names within the initrd (meaning when the Jetson is flashed with Seeed’s software or compared to when flashed with NVIDIA’s software), then you would just use Seeed’s kernel device tree (you’d always use Seeed’s device tree in boot stages prior to initrd).

The next part you might need to consider is if the ZED/Stereo Labs software adds any kernel modules. This part might require explanation if you find that the kernel itself is changed when using the ZED software, but I want to wait until we know that’s actually the case. If ZED/Stereo Labs has some sort of software to install into the software used to flash, then we can examine unpacking that software. Basically, I need more information now on how the ZED software is installed. I don’t know if it is its own flash software, I don’t know if it is a patch intended for NVIDIA JetPack, and I don’t know if it is run and installed directly to a running Jetson.

Now a similar look at .dtb files in the second log, log_zed_flash.txt (3.9 MB)

Looking at mentions of .dtb files in this second log I see this excerpt:

  • This more or less says we are interested in device trees here because they are kernel trees:
    Selecting previously unselected package nvidia-l4t-kernel-dtbs.
    Unpacking nvidia-l4t-kernel-dtbs (5.15.136-tegra-36.3.0-20240506102626)
  • Later it talks about the install steps for this. Mostly it has nothing else installed. Unfortunately, it doesn’t mention the header file name and only mentions the package. It is possible to copy a number of .dtbs into “/boot”, but not use them; there is also a possibility of putting a dtb into a partition as signed binary data. This in particular has me wondering:
    [ 0.0750 ] Kernel DTB used: None (maybe the first dtb package is installed but not used? Or perhaps configured later?)
    Similarly: [ 0.0750 ] WARNING: dce base dtb is not provided. It is also possible that the device tree used is inherited from the boot stage passing it on. In that case an actual serial console boot log would be used to find out what was loaded. In fact, a serial console boot log is quite important under both the ZED and the Seeed install methods because you can see what the boot is actually configured to use (be sure to remove “quiet” from any “/boot/extlinux/extlinux.conf” before creating a serial console boot log).
  • I now see these:
    • [ 0.2351 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0
    • [ 0.2830 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0 (note that this duplicates the prior log line)
    • [ 0.3261 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct.cfg --misc tegra234-mb1-bct-misc-p3767-0000_cpp.dtb --device tegra234-mb1-bct-device-p3767-0000_cpp.dtb
    • [ 0.3295 ] Parsing config file :tegra234-mb1-bct-device-p3767-0000_cpp.dtb
    • Those above lines might not matter as they refer to boot, but inheriting into the running kernel from boot stages is possible. I just don’t think that boot stage content would be important to a camera, and unless we see otherwise, those bct (boot control table) entries can be ignored. Most of the log goes on with uninteresting boot stage dtb logs.
    • This is where it gets interesting:
      Copy /home/vivio/Downloads/stereolabs_bsp/Linux_for_Tegra/kernel/dtb/tegra234-p3768-0000+p3767-0000-nv.dtb to /home/vivio/Downloads/stereolabs_bsp/Linux_for_Tegra/kernel/dtb/tegra234-p3768-0000+p3767-0000-nv.dtb.rec
      First, this is not a boot stage file. Second, it names the module/carrier board specification it is from (the p3768 and p3767 are module and carrier board specs, although I don’t know which ones off hand; the -0000 is just a revision number). It seems that p3768-0000+p3767-0000-nv.dtb is used by the Linux kernel even if it might get some rename after being copied into the ZED software. It is even useful that the log line lists where the files are on the host PC that flashes the Jetson. Comparing what ZED has here to what Seeed has for device tree might be interesting (and useful; if you see specific differences, then the ZED content can be added into the Seeed content and you’ll get a device tree which satisfies both vendors; caveat: assuming no conflicts).
    • I see a recovery image being built, and it uses the same “interesting” tree from above; the log line shows it uses the one which was copied in as Linux_for_Tegra/kernel/dtb/tegra234-p3768-0000+p3767-0000-nv.dtb.rec; Linux_for_Tegra/kernel/dtb/tegra234-p3768-0000+p3767-0000-nv.dtb.rec and Linux_for_Tegra/kernel/dtb/tegra234-p3768-0000+p3767-0000-nv.dtb are two names for the same file. ZED just uses a .rec extension.
    • Recovery images don’t much matter for our purposes. I don’t think the dtb mentions further down in that log are relevant. Note that bpmp also is part of boot stage, and any device tree with that in its name is not important to us.

So the gist is that we like to know about anything p3768-0000+p3767-0000-nv regardless of any file name extension.



Summary:

However, there is a twist here: The ZED software does later on use device tree overlays. These are the .dtbo files. Think of a device tree overlay as a patch to an existing part of a device tree. They are not used for adding tree nodes, but they can change them. In a way this is nice, because we know we have to compare those base device trees between that from ZED to that from Seeed, but an overlay is already separated out. In that case, if the edits are actually used (you’d check the serial console boot log with quiet removed), then you can directly use the ZED overlays on top of the Seeed device tree. This does require that the device tree nodes being edited already exist, but you’ll know that from decompiling the two trees of interest:

  • p3768-0000+p3767-0000-nv.dtb
  • tegra234-p3768-0000+p3767-0000-nv.dtb

So that is your next step: Compare those two. Attach them to the forum thread. I think they are more or less the same exact file. Then we find out if the ZED overlays can be added to the Seeed tegra234-p3768-0000+p3767-0000-nv.dtb (probably as a .dtbo…which we conveniently already have).

After we know how those differ (or do not differ) we can try flashing with the Seeed software and adding the overlays to the Seeed version.

1 Like

This is going to be a long war! Thank you for having you as an ally, @linuxdev

I have tested ZED/Stereolab method because after flashing nvidia SDK the camera had image noise on it especially on high resolutions. I wanted to see if it was the same. It might be some hardware problem here (maybe I need short connecting cables to CSI ports)

The ZED Orin box (the jetson box provided by Stereolabs) for sure (90% probability :) ) uses 2 CSI ports to connect to their capture card/camera.

If I may translate you substantial message:

  1. On ZED install method I have to edit AFTER install: remove the “quiet ” anywhere in “/boot/extlinux/extlinux.conf
  2. Use the serial log to see how the system boots up
  3. post the results
  4. Install with Seed method and redo the steps.

P.S.: I have seen a problem with 2>&1 | tee log_flash.txt> If I add to to a command like cd source the command is not recorded and does not enter on the device.
It might be some problem on my host device since I’ve added these days a new SSD and I’ve had some Read/Write access problems.

Ok, I have partial responses.

For new users: it seems that for serial connection in some cases might be needed a UART/TTL to USB device. Is not the one with the USB-C cable :(

On “/boot/extlinux/extlinux.conf ”, it was not “quiet ” anywhere fortunately. Clean as a whistle :)

Here it is the serial log with ZED/Stereolab flash method. Is almost stock, the drivers for camera WERE NOT installed. I thought it was easier to compare with the SEEED flash method.

This is the needed log?
ZED_like_UART_nodriver.log (76.5 KB)

Regarding your first reply:

  • The steps, including repeat, are correct.
  • You only need the “2>&1 | tee log_something.txt” for a command line that performs an actual flash. I don’t think that the cd commands will matter much. Use it on any command line where you want to attach a log to this forum.

Incidentally, I tend to save both a flash log and a boot log when I’m working with a new install. I might keep that for a few months. I don’t bother with that after I know things are working, and of course it only applies to command line flashes.

Thank you for confirmation. I’ll do the Seed version tomorrow when I’m back to the lab. The serial console was a pain in the … :))

At this point I know it is CSI, so that’s what I’m looking at. CSI is not hotplug, and so device tree matters. There is likely a GPIO being set up for i2c communications which is part of the CSI, but sort of “separate” in configuration.

It is correct that cables can have an extreme effect on high speed data lines. CSI qualifies as this, and as resolution goes up, the speed has to go up if the frame rate is to remain the same. So high res quality issues can (and in fact might be in your case) be the cause of lower quality. Cables can go as far as causing complete failure even if they are ok with DC electricity (AC, especially as the frequency goes high, has issues with reflections; this is more of a problem with square waves used in digital lines). You’d need to start a different thread on CSI cables (I know RF engineering, but CSI and cameras are not my field).


A serial console boot log cares about “quiet” being removed, whereas flash logs won’t be affected by this.

Just commenting on things as I go through the ZED flash log…

I see some notes of PMIC rail info not being found. This is the power management IC. There are different rails for different parts of the system, including i2c rails and rails which might affect cameras over CSI. This could cause a failure of a component simply because a device tree is not locating the item (or the rail that powers the item). Sometimes there are more rails being tested in flash than might actually exist on a model, so it might not always be an error. In addition to this, I see the information about being missing is in the MB1 BCT (master boot 1 is very early boot; BCT is the boot control table), but then it says “DONE”. Example log lines:

[0000.911] W> PMIC_CONFIG: Rail: GPU rail info not found in MB1 BCT.
[0000.917] I> DONE: GPU rail info
[0000.921] W> PMIC_CONFIG: Rail: CV rail info not found in MB1 BCT.
[0000.927] I> DONE: CV rail info

Perhaps config worked from some future step? Don’t know, but in part we are in the realm of what a device tree might solve…finding rails or parts of the system which are general purpose and might move (physical address might move).

Further down here are some similar items, but they were successes ():

0001.248] I> Binary name: BPMP_FW_DTB
[0001.252] I> Size of crypto header is 8192
[0001.256] I> Size of crypto header is 8192
[0001.260] I> strt_pg_num(13056) num_of_pgs(16) read_buf(0x807fc0000001.248] I> Binary name: BPMP_FW_DTB
[0001.252] I> Size of crypto header is 8192
[0001.256] I> Size of crypto header is 8192
[0001.260] I> strt_pg_num(13056) num_of_pgs(16) read_buf(0x807fc000)
[0001.266] I> BCH of BPMP_FW_DTB read from storage
[0001.270] I> BCH address is : 0x807fc000
[0001.274] I> BPMP_FW_DTB header integrity check is success)
[0001.266] I> BCH of BPMP_FW_DTB read from storage
[0001.270] I> BCH address is : 0x807fc000
[0001.274] I> BPMP_FW_DTB header integrity check is success
[0001.280] I> Binary magic in BCH component 0 is BPMD

If you look closely at this, this is a case of a device tree fragment for the BPMP (boot power management processor) which is being read from a partition as binary data. This was added to a partition via flash, and is an alternate location for such content. Assuming security fuses were not burned, a device tree which is accessible in /boot, and configured to be used with extlinux.conf, would take priority over this. Normally (without security fuses burned) this would only be read from the partition if (A) extlinux.conf did not say to use a device tree in a file of /boot, or if (B) the file named is not found. If the boot stage is too early to access extlinux.conf, then content might be read there anyway (this qualifies as extlinux.conf not specifying a device tree). Perhaps what is in the partition is correct only for NVIDIA dev kits, but perhaps only tiny parts of this are wrong on your Seeed carrier board. The parts (fragments) which are correct would work.

Now I’m looking for signs of a device tree file being loaded, rather than a partition being loaded. In particular it would be useful to see what it says is loaded for the Linux kernel rather than from early boot stages, or anything that might be related to the camera or CSI.

I see:

[     5.819549] Camera-FW on t234-rce-safe ready SHA1=e2238c99 (crt 0.891 ms, total boot 48.238 ms)

This might just be an early stage setup and might not matter, but if it does matter, then this is a big deal:

E/LD:   init_elf:486 sys_open_ta_bin(bc50d971-d4c9-42c4-82cb-343fb7f37896)
E/TC:?? 00 ldelf_init_with_ldelf:131 ldelf failed with res: 0xffff000c

I don’t know enough about this software to say if this matters, and there is more than one way to start the linker, but at this early part of boot a lack of ELF libraries being able to load (ld is the linker) only statically linked programs would succeed. Everything depending on a library would fail. The log is not done yet though, so maybe further down there are clues of success. Incidentally, boot stages are always bare metal, and do not link; the Linux kernel itself is also bare metal and works without linking. This point in the logs is right at the start of changing from boot content to Linux kernel, and so probably this does not matter.

Sadly, these log lines would be more useful if it gave actual file names:

L4TLauncher: Attempting Direct Boot
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Exiting boot services...

I see no further mention of device tree (addresses mentioned for things like i2c would be from the earlier device tree, but that might be the one in the partition).

This might say that power management is set up wrong:
[ 9.121659] ACPI: <n/a>: failed to evaluate _DSM (0x1001)


The good news is that it does boot and allow login. I did not see anything particularly relevant to the camera.

The detail to remember while continuing on is that the device tree that is put into a partition during flash is the one which is used. On such a system we could add a file type .dtb in /boot, modify extlinux.conf, and test without flashing again. Once we see the same thing on the Seeed flash and boot log we will know about what device tree is used there. Then we can try to find the device tree used by ZED for the partition, and compare. If there is a difference related to CSI, then that can be edited, and then the Seeed version might work (you’d want to save a safe copy of that edited tree, or at least the fragment of the tree being edited).

Even so, I did not see a lot of failures on the ZED version. Is the only reason you use a particular release (Seeed versus ZED) due to the high resolution settings having a low image quality?

Once again, a wonderful situation in which is cool to learn something new!

Very clear explanation, although some fundamental background is needed on my part.

The artefacts camera I’m not sure if were present at first. It might me just a simple bad connection duet to the fact I’ve mounted/dismounted CSI cables.

My findings:

  • JP5.1.1. SEEED flash + normal drivers for capture card: Camera OK
  • JP5.1.1. SEEED flash + drivers for ZED Box ORin with capture card: Camera FAIL
  • JP6 SEEED flash + normal drivers for capture card: Camera FAIL
  • JP6 SEEED flash + drivers for ZED Box ORin with capture card: Camera FAIL
  • JP6 Stereolabs flash + normal drivers for capture card: Camera FAIL
  • JP6 Stereolabs flash + drivers for ZED Box ORin with capture card: Camera OK
  • JP6 nvidia sdk + normal drivers for capture card: Camera FAIL
  • JP6 nvidia sdk + drivers for ZED Box ORin with capture card: Camera OK

So, basically the camera works on JP6 only on Stereolabs flash or nvidia sdk, but with drivers developed for ZED Box ORIn that has the captured card inside.

Only on JP5.1.1 the components have played nicely… SEEED install on J4012 and from Stereolabs just drivers, since I have just the capture card and camera from them. I needed access to the carrier board and that’s why I did not bought whole solution from Stereolabs (jetson+capture_card+camera)

The main reason of messing with those custom flashes is to have a real functional board on which to have vision from Stereolabs and be able to run a newer Ubuntu needed for ROS2 (robot operating system)

For the working cases:

  • JP5.1.1. SEEED flash + normal drivers for capture card: Camera OK
  • JP6 Stereolabs flash + drivers for ZED Box ORin with capture card: Camera OK
  • JP6 nvidia sdk + drivers for ZED Box ORin with capture card: Camera OK

Note: Any capture card which is not part of the Jetson itself needs to be described. Capture cards have their own drivers and setup. How any capture card is connected also matters, e.g., if it is USB or PCIe. If the manufacturer of a separate capture card is Stereo Labs, versus some other manufacturer, that also needs to be described.

…when you say the camera is OK, is this the ZED camera which you are saying is ok? Or is there another camera involved? If the ZED camera works in those cases, what is missing that you need to change or add? Is there a separate capture card which comes with the ZED camera? My assumption has been that there is only a ZED attached to the integrated CSI of a Jetson.

Incidentally, I’m not a “Docker guy”, but many people use Docker to work with ROS2 (you’d need to start a new thread on that).