Looking at the first bot error logs, seems like its triying to mount some volume and failing. The first flash logs looks ok. Something easy to try is to change the sd to a known good one. @linuxdev maybe has more insight on the boot error logs.
Would it be possible to attach a complete serial log file rather than copy and paste? Whatever goes on might be related to earlier boot.
I’m new on this thread, and the first thing I wonder is the specific Jetson model you are using. Is this a Nano dev kit with SD card slot and no eMMC? Or is this a third party carrier board with a Nano that has eMMC? Is there any other external storage device involved? Consider whatever I’m saying to be from a new perspective that has not followed this thread.
I don’t know about your specific case, but it wouldn’t be unusual for module load to fail if (A) the filesystem type is not supported that the modules live on, or (B) the modules are being searched for on the wrong device, and although not likely from that message, if the modules were compiled against a different kernel configuration and not valid in this configuration.
In the case of a custom kernel, did you get the source code which goes with this particular L4T release? You can find the L4T release with “head -n 1 /etc/nv_tegra_release”. I would also be interested in knowing the L4T release.
In the case of any kernel build, what was your goal? Which symbol or change are you interested in? How have you installed any kernel or module in the case of customizing it?
1 Like
Hello, thank you for your response. I try to summarize the situation and the hardware components.
Also attached the complete serial log from minicom:
minicom.log (34.3 KB)
-
Jetson model: I have the official Nano dev kit with (32 GB) SD card and also a third party carrier board with soldered eMMC. For the host machine I am using an ubuntu 18.04. At the start I tried both of the boards with similar outputs, but in the last 2 weeks I only worked with the 3rd party eMMC version.
-
Versions: Yes I have to admit that now I am a bit confused about the board versioning, after the failed flashing I used the SDK Manager to ref-lash the boards and for the 3rd party board I chose the P3448-0002 option (based on the vendor tutorial). In the case of the RidgeRun guide I used the tegra210-p3448-0002-p3449-0000-b00.dtb and the following command to flash the image sudo ./flash.sh jetson-nano-emmc mmcblk0p1. During the manual flash I used the 32.7.1 and 32.7.4.
-
L4T version: After a clean flash with the SDK Manager I have the following: #R32 , REVISION: 7.4 BOARD: t210ref, EABI aarch64 for both boards.
-
Goals: I tried to implement a driver for an ov5647 camera. I implemented a ov5647.c file and v5647_mode_tbls.h file and modified the on board DTB fila after decompile into dts file.
-
Installs: At first I added my files next into the kernel/nvidia/drivers/media/i2c/ folder modified the Makefile and the Kconfig file, then modified the /kernel/kernel-4.9/arch/arm64/configs/tegra_defconfig file to include my driver CONFIG_VIDEO_OV5693=y. I added my device tree file. After that I followed the RidgeRun guide and run the make commands. I had my files compiled with the original ones, but when I tried to boot my device I reached the not booting problem. After multiple attempts, I tried to follow the guide without modification of the original files. And that was the moment I had to realize I making a mistake somewhere the process because I had the same non-booting problem with the default source files also. I also tried to use SDK manager with the manually compiled Image file and dtb file but i gave the same output(non-booting.)
I attached the log from the manual flash:
ridgerun_flash.txt (18.0 KB)
I also tried to manually reflash just the dtb after the Image flash this is the the log about it:
ridgerun_dtb_flash.txt (16.7 KB)
The soldered eMMC is very suspicious as an issue. Is there published information for this? Basically any hardware added which is not plug-n-play means the device tree has to be updated, and possibly drivers need updating. eMMC in particular needs boot stage edits as well. A lot more needs to be known about this soldered eMMC, especially if the manufacturer provides information on it.
How this eMMC is to be used also matters. If this is originally an SD card model (SD slot on the module, not on the carrier board), then it takes a lot of software modification to boot properly from something other than the SD card (you could do this via an initrd boot for example).
In the boot log it gets to the stage of starting Linux like this (an excerpt of the log lines):
copying carveout for /host1x@50000000/dc@54240000...
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0
The fact is that it doesn’t state what the kernel’s arguments are. This seems more like a UEFI boot, but it is still the standard Nano CBoot/UBoot, so I am confused why it didn’t name the command line arguments. It seems modified, which might or might not be an issue.
The third party carrier board definitely needs modifications to device tree (firmware). Use of eMMC during boot for an SD card model on that carrier board also definitely needs a lot of software modifications.
All of the above modifications have to come from the company which designed and sold this carrier board. It is used in combination with a Jetson module. Beware that they may be expecting a module which does not have an SD card slot, but has eMMC, or it might be for the SD-card-on-module model, but that would be odd for a number of reason. Mostly, what does the manufacturer provide for flash software? Does that manufacturer provide software and their own documentation?
Now so far as the kernel driver goes, you don’t need to flash to add that. If the camera uses some methods of connecting, then it is more or less plug-n-play, and wouldn’t even need a device tree edit. I’m going to suggest you get the board working without the driver to start with. Then come back and get help on driver install.
If this is a RidgeRun board, then you are in luck because they have all of the software you need and information. We could then help with driver install. If this is not a RidgeRun carrier board, then that guide is probably not quite correct since the hardware would differ (and thus so too would the firmware differ).
Thank you for your response. You were right, I checked the vendor site of the 3rd party board and it had a guide to flash it but in that case the source files are already compiled and I was not able to add my driver sources.
But based on your post I turned to the official NVIDIA SD card based model to avoid any modification or 3rs party based problem.
Based on the Camera Development Guide I thought I can not avoid kernel rebuild to use my driver.
But based on your suggestion I continued with a non-kernel-building way. I built the driver files on the board and load them with insmod the .ko files, this has no effect on the board booted normally but no sign of the ov5693 sensor detection. I used dmesg and kern.log but it only contained one line:
ov5647: loading out-of-tree module taints kernel
After that I used the modified version of the default device tree which was in the /boot/dtb/kernel_tegra210-p3448-0002-p3449-0000-b00.dtb. At this point I tried to remove all the imx219 related parts and replace with the ov5647 related parameters. After rebooting the device I still not see any effect of my modifications, even the imx219 camera still operated. At this point I was suspicious because I removed the imx219 related parts from the device tree, so I repeated the modifications with the boot/kernel_tegra210-p3448-0002-p3449-0000-b00dtb and the /boot/kernel_tegra210-p3448-0000-p3449-0000-b00.dtb files. after rebooting the board the imx219 enumerated as i2c device and worked properly. Now I am confused how is it possible, I tried to localize the used dtb file via dmesg and kern.log and boot messages via TTL reader log but I did not find it. It looks like does not matter if I modify a dtb file in the boot or boot/dtb folder it has no effect on the booting.
When I reached this point I turned again to the RidgeRun guide but with the NVIDIA SD card based model
official NVIDIA guide for kernel customization but I still stuck at the following error message during the build:
I want to understand the process and the problems, I have the logs files and followed the debugging steps, but when I try to follow a guide step-by-step always get stuck at a certain point.
For example if its an official nvidia jetson carrier board I can say it’s a ridgerun board? If it is when can not working the kernel building guide.
At the start I thought I understand correctly the relationship between the boards and the software versions, but now I am a bit confused. I am using the jetpack 4.6.4 from sdk manager, but based on the RidgeRun guide I downloaded r32-3-1_Release_v1.0/Sources/T210/public_sources.tbz2 with the gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz I have a 64 bit ubuntu 18.04 as host system and a Jetson Nano developer kit with the model name 3450 with SD card slot. I checked the type with the
cat /sys/module/tegra_fuse/parameters/tegra_chip_id
Now I am started to repeat my previous actions, and not sure how to move forward.
I’ve never worked with camera sensors, but I know there is more than one interface which might be used. How does your ov5647 physically connect to your Jetson? Some of those methods might be plug-n-play, e.g., USB or PCI, but other methods will require the device tree to be modified to tell the driver where and how to find the device. If the driver is present, then the other half of the equation is knowing where it is (basically a physical address and any arguments needed for passing to the driver). Plug-n-play can self-describe, other components cannot.
An example is that a USB device might follow the USB Video Standard (UVC). The self-description on plugin tells the driver everything it needs (with a little help from udev). This would produce a “/dev/video#”, which in turn is really the driver being bound to the hardware. Other types probably don’t produce anything unless that device tree is correct. It is hard for me to be helpful since I’ve not programmed anything with a camera sensor.
If you do have a module, then upon insmod any hardware should show up. Something which might be useful is to monitor “dmesg --follow”, and then watch what shows up as a result of “insmod” for your module. The log lines appearing as the result of a module plugin can reveal useful information (and so too can a lack of log lines reveal something useful). Note that after you have a module in place you probably also want to run “sudo depmod -a” because the next reboot could miss seeing this if the module isn’t registered.
An additional test to consider is that a module which is not bound to hardware has the ability to be dynamically removed via “rmmod” of that module. If the hardware is bound to it, and if there are dependencies, then rmmod of that module will fail. It would be useful after the insmod to see if rmmod works, or if it is refused.
Please note that a third party carrier board needs a different device tree than the NVIDIA carrier board. An eMMC model module, even if on an NVIDIA carrier board, will also need a different device tree due to the module itself being different. I’m very suspicious of your device tree and the mixing of RidgeRun and NVIDIA device trees and source code. This really needs to be validated before you can say much at all about the actual issue you are working on.
It is possible that the RidgeRun kernel uses the same source as the NVIDIA one if it is based on the same L4T release. You can check L4T release with:
head -n 1 /etc/nv_tegra_release
What L4T release are you using? Is there a web page you can point out with the exact Jetson you are using with RidgeRun?
If this is RidgeRun hardware, then someone on this forum might be from RidgeRun and able to comment as well.
I have some experience with UVC, I developed one firmware for a UVC camera. But this time I have a MIPI camera and I attached it into the CSI port and tried to follow the guides and the similar forum posts. I got great help from @andres.artavia, but after the multiple failed kernel flashing you were summon.
I am asking your opinion about the kernel recompiling problem. In the case if I use the Offical Nvidia Jetson Nano Developer KIT I not remember the site I purchased the board but it’s an original SD card version with 2 CSI port. Model number P3450.
I checked the JetPack folders and it used the 32.7.4 version now I restarted the guides with these version. You also mentioned the possibility of avoid kernel recompiling, how can I compile driver files on the jetson board itself?
Also what do you think about the device tree phenomena I mentioned it looks like my modifications has no effect on the board’s behavior.
Hey @WayneWWW do you have any pointers to what he can try?
Most kernel and kernel module installations do not require flashing. Similar for device tree edits. If something is not needed during boot, then it is simpler than some change which must exist while boot is running. My guess is that you can use a simplified kernel and/or module install method (and perhaps device tree install) and then concentrate on finding out if the kernel actually installed.
In the case of adding drivers you can often add as a module and simply copy a file.
If you build an integrated feature (not a module; module uses “=m” in config, integrated uses “=y”), then it gets more complicated, especially if you use initrd boot. I’ve never worked on that particular driver, but if the driver is available as a module, then you might be able to simplify things by building as a module.
Keep in mind that it is ok to directly edit a config setting if and only if you know that the driver has no dependencies. If you do not know that answer, then it is safer to use a dependency-aware tool for config editing, e.g., the make target of “nconfig” or “menuconfig”.
What is your current kernel actually seeing? What is the result of this command?
zcat /proc/config.gz | grep 'OV5693'
If that shows up as either “=m” or “=y”, then your kernel update succeeded. If the sensor is not plug-n-play, then the only step left is device tree. If the device itself is not using a plug-n-play interface, then you’ll probably have more work ahead in order for programs or drivers to find the device. Not all video devices will produce a “/dev/video#” entry, and so you might need to look at some other “/dev” file. I simply don’t know enough about that device.
I achieved some progress, I started over with the 32.7.4 sources and the corresponding kernel customization guide. I followed @andres.artavia’s guidance.
I flashed the Jetson Dev-kit via SDK manager with my driver files. Now the board booting without any problem and if I type in the following command :
zcat /proc/config.gz | grep ‘OV5647’
It returns with the content of the modified tegra_defconfig file:
CONFIG_VIDEO_OV5647=y
During building the sources I also included the my modified device tree-file, but after the flash and the boot still no sign the camera (except if I check the config) so I checked the device tree file in the boot folder at it looks like the default. I tried to replace it with my modified version and reboot but still no change.
Now I am able to build the kernel and flesh the board, and read more similar posts and try to understand what’s happening. Please correct me if I am wrong:
- With the result (CONFIG_VIDEO_OV5647=y ) I can say my driver files are part of the kernel
- If my extlinuxconf file does not contain a FDT part I can add one to point to my modified dtb file?
- If I want to test and debug my camera driver I should modify the default device tree or I should add an overlay file with my camera related stuff?
Did you also rebuild and install all kernel modules? Since you changed the Image with an “=y” parameter you should do this (and pick a new module install location via a new CONFIG_LOCALVERSION; default is “-tegra”, so you could change it to something like “-test” or “-ov5647”; you could dual boot this instead of immediately overwriting the original in case it doesn’t work).
Whenever something like this fails you should post a full serial console boot log.
You can add an FDT line to extlinux.conf. This will take priority over the device tree partition which is created during a flash. The exception is that if you have security fuses, and have burned those fuses, then /boot content is ignored, but a Nano does not have security fuses to burn.
Once booted with a new kernel also always run the command “uname -r” and see if your modules are located there (“/lib/modules/$(uname -r)/”). The suffix of that command should be your CONFIG_LOCALVERSION.
I achieved new progress, I modified the extlinux.conf file to point my modified device tree and now I can find “OV5647” related logs with dmesg:
[ 1.380084] ov5647 7-0036: probing v4l2 sensor.
[ 1.380087] ov5647 7-0036: i2c_client->name = ov5647
[ 1.380090] ov5647 7-0036: i2c_client->dev = (null)
[ 1.380094] ov5647 7-0036: node = (ptrval)
[ 1.380128] ov5647 7-0036: Failed to find clocks
[ 1.380132] ov5647 7-0036: unable to get platform data
[ 1.380135] ov5647 7-0036: tegra camera driver registration failed
[ 1.380168] ov5647: probe of 7-0036 failed with error -14
[ 1.380208] ov5647 8-0036: probing v4l2 sensor.
[ 1.380211] ov5647 8-0036: i2c_client->name = ov5647
[ 1.380214] ov5647 8-0036: i2c_client->dev = (null)
[ 1.380217] ov5647 8-0036: node = (ptrval)
[ 1.380247] ov5647 8-0036: Failed to find clocks
[ 1.380250] ov5647 8-0036: unable to get platform data
[ 1.380253] ov5647 8-0036: tegra camera driver registration failed
[ 1.380284] ov5647: probe of 8-0036 failed with error -14
I am so exited that now I have any sign for the presence of my driver file. I was not sure about the eeprom of my camera, so based on other posts I will start with disabling the plugin manager to use pure I2C.
I think I reached the point what you mentioned in your response on Jan 30 @linuxdev . I am also grateful for your previous guidance @andres.artavia do you have other advices for the current situation?
Hi,
Happy to hear the driver probe starts. Usually the template or base drivers have more gpio/clks/voltage regulators checks that what’s actually used on the hw, you can check the probe method flow to verify that is checking only what you need and have on the device tree. And also you can remove error returns, so for example on some drivers you have
if (something_not_ok){
dev_err("Error print");
goto error;
}
...
error:
return error_n;
and you can change it to:
if (something_not_ok){
dev_err("Error print");
}
...
error:
return error_n;
so it just prints as a warning instead of exiting.
That sounds to me like thats the issue, either some property is missing on the dtb or an extra check that might not be needeed is being done that is making the driver probe fail.
Regards,
Andres
Hi,
I continued the exploration, I recompiled the kernel one more time but, now I set the ov5647 “=m” instead of “=y”. I thought I could use insmod to testing new driver versions without kernel building. I also added some printing lines to my driver functions but I got back the same error messages.
I also tried different device tree iterations, but now change in the status messages. But I also not see the print messages, where should I find them?
I re-read the sensor register part of the guide, but now I am a bit confused about what should I do with the plugin manager. I read the link you attached about the overlay files but to be honest I am not getting why better the dtbo file than the default device tree file modification.
Usually what I do is to replace the warn and debug prints with dev_err(), so its easier to identify them on the dmesg, and then you can do:
sudo dmesg | grep ov5647
And regarding the dtb, it seems like some of your device is being added, since if it was not properly added on the dtb, the probe won’t even run. So on there you are on the right track.
The message:
unable to get platform data
Sounds to me like something on the device tree is not found. Take for example the imx219 driver:
priv->pdata = imx214_parse_dt(client);
if (!priv->pdata) {
dev_err(dev, "unable to get platform data\n");
return -EFAULT;
}
The error comes from the method:
static struct camera_common_pdata *imx214_parse_dt(struct i2c_client *client)
{
struct device_node *np = client->dev.of_node;
struct camera_common_pdata *board_priv_pdata;
const struct of_device_id *match;
match = of_match_device(imx214_of_match, &client->dev);
if (!match) {
dev_err(&client->dev, "Failed to find matching dt id\n");
return NULL;
}
board_priv_pdata = devm_kzalloc(&client->dev,
sizeof(*board_priv_pdata), GFP_KERNEL);
if (!board_priv_pdata) {
dev_err(&client->dev, "Failed to allocate pdata\n");
return NULL;
}
of_property_read_string(np, "mclk", &board_priv_pdata->mclk_name);
board_priv_pdata->pwdn_gpio = of_get_named_gpio(np, "pwdn-gpios", 0);
board_priv_pdata->reset_gpio = of_get_named_gpio(np, "reset-gpios", 0);
board_priv_pdata->af_gpio = of_get_named_gpio(np, "af-gpios", 0);
of_property_read_string(np, "avdd-reg",
&board_priv_pdata->regulators.avdd);
of_property_read_string(np, "dvdd-reg",
&board_priv_pdata->regulators.dvdd);
of_property_read_string(np, "iovdd-reg",
&board_priv_pdata->regulators.iovdd);
return board_priv_pdata;
}
So for example the default imx219 had more checks that our setup didn’t need but was making the probe fail. So we just removed them. You can check the name of the device tree property that it wants and either add it or remove the return if fail statement.
You can use device tree decompiler to check the currently running tree to file and verify all you added is there, check here. Those things are what comes to mind on you can check.
Regards,
Andres
I am still working with the driver now I was able to add more debug message. But I want a ask an other part of the kernel building. I read multiple posts with similar problems/questions. I tried to turn off the plugin manager and add my dtsi file.
For that based on the posts I edited files in the:
nvidia\platform\t210\porg\kernel-dts folder i removed the plugin manager related includes and added my alternative device tree file also added to the make file. Built the files and I could check that my dts file dtb version is compiled, but after flashing the board I checked the boot folder and did not find the custom dtb file.
I could modify the extlinux.conf file and manually added my dtb file and after that the board tried to use my driver.
But the plugin manager is still active and I have a terrible feeling it could affect the working of the probing process.
Also looks strange to me that my dtb file missing from the board.
Regards,
Bence
My original problem was resolved, know I can modify the device tree and able to compile the kernel, so I will accept as a solution. And the driver development a different question. Thank you for your help!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.