Orin won’t complete boot successfully and I was unable to put it into recovery mode either (it seems to loop). Since this is urgent could we set up a call?
(I am also unable to to connect a host pc to it, in case I’d need to flush with the SDK Manager. That seems like a cable problem)
Thank you!
I can’t work on a telephone, but here is some information…
You should state if this is a developer’s kit AGX Orin, versus some third party carrier board and separate module. If you’ve flashed, then you should say what you flashed with (which version). Specific release content and documentation will be for your L4T release or JetPack release (L4T is Ubuntu plus NVIDIA drivers, and this is what gets flashed; JetPack/SDK Manager is a GUI front end). For flashing you would want a native Ubuntu 20 host PC (though Ubuntu 18 works; other releases probably work if using command line). VMs tend to fail due to needing USB setup. If you try to detect a recovery mode Jetson via a VM on a host PC, then there is a very high chance you won’t find it unless USB is set up correctly.
The recovery mode button is like a shift key on a keyboard when trying to get a capital letter. However, the recovery button acts on either power on or power reset. To be in recovery mode you would hold the recovery button down, and tap either the power on or power reset, then let go (there is no requirement to hold the recovery button down).
On the host PC you can see all NVIDIA USB devices via:
lsusb -d '0955:'
An Orin would have device ID 7023, and so you’d see it with:
lsusb -d '0955:7023'
The L4T release is found via “head -n 1 /etc/nv_tegra_release”. If you’ve installed the flash software onto the host PC, then this is found at “~/nvidia/nvidia_sdk/JetPack...version.../Linux_for_Tegra/”. You can find what L4T release this would flash from the host PC if you cd to the “Linux_for_Tegra/” directory, and then:
cd rootfs/etc/
head -n 1 nv_tegra_release
You would always use the newest JetPack/SDKM, but if you need it to flash an earlier release, then you can start sdkmanager like this (use your regular user, not sudo):
sdkmanager --archivedversions
The URLs to find a given release of L4T or JetPack (their versions are tied together) are here:
A checklist of questions:
gtkterm, but a lot of people use minicom) can log. Get the entire boot. Quite often the system is booting fine, but the GUI is not working, and people mistake this for a failed boot (which is worse if you use anything Wi-Fi since normally Wi-Fi requires a user login to the GUI). If the serial console does work, then you can log in there and rescue the system.This is great information. This is the Orin AGX that comes in the gray metal casing. I will provide all info as soon as I get back, I’m driving now to try and find a proper usb cable, as I must have tried 5 types of usb A to C already, so the immediate question is what kind of a cable is this. Should the Orin side be a USB 2 and the side that goes to the host USB 3 ? (in case you see this before I get to Best Buy… 🙂)
Will this work?
Have you tried to flash?
Yes I was trying to flash but can’t get the sdk manager or lsusb to find the usb.
Which release? I was on r35.2.1 and was trying to upgrade to r35.4.1
Are you using a VM? Not that I know of.
Is this purely a developer’s kit? It’s the Orin AGX with gray casing
Hi,
I think when you have tried 5 different cables with all of them not working, then it’s not the issue of cable you use, but whether the device itself is dead, or you don’t know how to put it into force recovery mode correctly.
Ensure that the developer kit is powered off.
Press and hold down the Force Recovery button.
Press, then release the Power button.
Release the Force Recovery button.
Did you follow these steps?
It is an AGX Orin Developer Kit, and the force recovery button is the middle one on the side of the device, next to the microSD slot…
@DaveYYY I just followed your force recovery power-up instruction exactly (middle button of the three) and what happens (as I have seen before) that it cycles:
Nvidia logo in Green and Nivida title below and the bar bar that fills up to the right
text lines sent to stdout (like “scroll credits” that scrolls too )
Black Screen
Return to Nvidia logo in green…
repeat…
Then can you see this?
What can you get from the serial console?
let me try again although I didn’t see anything added before on lsusb…
no. nothing shows up on lsusb -d ‘0955:’
I haven’t tried serial console yet. I’ll do this next. But I I don’t have the the port ID ?
I am not comfortable with minicom, I’ll install gtkterm
Only the USB-C port on the front side of the device, next to the 40-pin header, can be used for flashing.
Did you use the right one?
yes. (opposite side to where the USB ports are on the same panel as the power)
I installed gtkterms and I’m trying to figure out how to use it. Is there a quick way to find the port? right now I still can’t identify it.
You should use the microUSB port for capturing serial console log.
It will appear as /dev/ttyACM0 on your host PC after power on.
Thank you! I Will try first thing tomorrow, as it got later here
I just got back to this, and wanted to add some more information.
What you have is the NVIDIA developer kit. This is good, it simplifies some things. I actually have one that I think was a pre-release, before it hit the market. So it is possible mine differs in some details, but I don’t know if that is the case or not.
I mention because on most earlier Jetson models (not Orin), in which the serial UART was a dedicated USB port (before that it was two pins exposed to TX and RX of a UART), that this was a micro-OTG USB port to the left side of the 40-pin header (2x20 arrangement). This is what provided the serial console. On my AGX Orin this transitioned to a USB-C. The older micro-OTG used a micro-B cable for signal (often sold as a “charger” cable), but the AGX Orin I have uses USB-C for that. If you have a micro-B “charger” cable, then the cable itself is always suspect as cell phones and such charged through such a cable typically have no more than 2 strands of very tiny copper (e.g., 2 or fewer strands of 32 gauge or thinner wire) for the data part. USB-C does not have that issue. The micro-B cable when NVIDIA supplies on earlier models is of high quality and works well with sustained higher speed transfers.
If you use gtkterm for the program (and I don’t care for minicom myself since minicom has a lot of “AT command” support for modems from the 1980s or similar modems that we just don’t use anymore), and this complicates it. gtkterm does have the disadvantage though of requiring a GUI, whereas minicom does not require that. Assuming that the serial port produces “/dev/ttyACM0” on your host PC upon plugin to the USB port to the left of the 40-pin header, this command will view the port properly:
gtkterm -b 8 -t 1 -s 115200 -p /dev/ttyACM0
(you might need to use sudo)
If on your host PC you want your regular users to be able to watch serial console without sudo, then add your PC user to the group “dialout” like this (I am assuming your login name is “user”, adjust for your actual user login name):
sudo usermod -aG dialout user
Note that USB is hot plug, and if the USB UART is powered (either by plugin or by the Jetson, I’ve never checked if the UART needs the Jetson powered…most UARTs are powered by the host plugging in the USB end, but not always), then log messages will occur as a result of plugin. If, on the host PC, you monitor “dmesg --follow”, then the USB plugin will show a log message. Part of that log message is about the USB pipe itself. Seeing any activity in logs upon plugin verifies the USB side is probably working (unless the log message actually says it is failing).
The USB side itself does not generate the device special file (“/dev/ttyACM0” is a device special file…it isn’t a real file, it does not exist on the disk, but is instead in RAM as part of the driver and pretends to be a file). When the driver loads there should be another message in the logs…this is the hot plug telling you it associated a driver with the data pipe for the end hardware (versus for the data pipe). If you see a log message that includes ttyACM0, then your serial console is ready for gtkterm (or any other serial console program at 115200 8N1 settings, no hardware flow control). The creation or existence of that file implies the driver has loaded for the serial UART and is currently running. The lack of this file suggests no driver loaded, and will occur if either (A) the driver does not exist, perhaps needing install, or (B) the USB pipe has an error.
Be sure to start gtkterm logging before you apply power to the Jetson (you might need to be sure of the device special file name first, but consider that a test run) if you intend to provide those logs to the forum. gtkterm might make it so you have to use an existing file name for logging, but you can create an empty file like this (I have directory “~/Dowloads/deleteme/”, which is where I put stuff that is temporary when I expect it needs to be deleted; use any location you want):
touch ~/Downloads/deleteme/jetson_boot_log.txt
OR:
touch ~/Downloads/deleteme/recovery_mode_log.txt
The flash port will be the USB-C cable on the other side away from the 40-pin header. I use a cable provided by NVIDIA which came with the Jetson for the USB-C (the other end of it is USB-A). I use a micro-B USB cable for the serial UART console which is also provided by NVIDIA, but I think it arrived with one of my other Jetsons. Most USB-C will suffice, but be very suspicious of any charger cable (I would say two out of three charger cable designs will fail for sustained higher speed data).
An important thing to know, which most people don’t need to use, is the fact that the serial console is also useful for debugging during a flash. During a flash the jetson does not have login capability (it’s just a custom USB device, and login is from Linux which does not run on the Jetson during a flash until flash completes and the system automatically reboots to a non-recovery mode). However, sometimes if there is an error during flash it is useful to have that log as well (we differentiate this as a flash log, versus a boot log when we boot normally). Just knowing there is no serial console activity during a flash is itself a clue.
I want to also add that for the USB-C I flash via a USB type-A port as well, but it is USB3. I think a USB2 port should be ok as well, though taking longer. Sometimes though USB3 is actually required for some devices. I doubt that the AGX Orin requires USB3 and works with USB2 so long as it is in recovery mode. Just watch the “dmesg --follow” on the host PC with the USB-C to type-A cable connected, and see if any message is visible. You’ll only see messages about the USB pipe itself since Jetsons have a custom driver (the driver is actually the flash software).
I will say it is possible that an adapter between USB-C and type-A will cause a failure if the signal quality drops, but it is unlikely to fail. It “should” work, but it isn’t a guarantee.
Wow! Thank you for such a wonderful comprehensive explanation! by far one of the best remote guidance I have ever received. 🙂 I truly appreciate this! ok. I will dig deep into what you are saying. There is some learning for me here, but It’s encouraging to know that serial communication is not that different from other platforms. I spent a bunch of time on arduino microcontrollers and have some familiarity working with serial consoles, AT commands, and tx/rx, so that will come handy… Also, I previously used the micro-usb to USB A on the Xavier NX to flash through the SDK manager. This is btw, why I got suspicious of the cable on the Orin AGX. It had to be just the right one on Xavier NX. Let’s see what happens in the following hours…
Hi @linuxdev
Here is the output from the serial console. Please not that this was generated on the gtkterm after connecting with a micro-usb - usb A on /dev/tty/ACM0 port like you recommended, and following a force recovery steps. A usb-C - usb-A wasn’t connected
recovery_mode_log.txt (97.9 KB)
Hi,
The log indicates there are some problems in your “upgrade”.
In brief, your upgrade is not completed.
The device tree timestamp is Jan 24, 2023.
[ 0.005298] DTS File Name: /dvs/git/dirty/git-master_linux/kernel/kernel-5.10/arch/arm64/boot/dts/../../../../../../hardware/nvidia/platform/t23x/concord/kernel-dts/tegra234-p3701-0000-p3737-0000.dts
[ 0.005308] DTB Build time: Jan 24 2023 15:26:09
But jetpack 5.1.2 is released on 8/1, 2023. Which means some of your software is from old release while other of them are from new release… this leads to crash…
BTW, I don’t know what did you do before. My comment is under the assumption that it looks like you still not flashed the board yet.
No. I didn’t flash yet. At-least not intentionally or in any way that I am aware of.
I do recall that an error related to nvidia-utils-525, and
sudo apt --fix-broken install would complete with an error which the initial reason I was recommended to upgrade to 5.1.2. I believe that error was:
Errors were encountered while processing:
** /var/cache/apt/archives/nvidia-l4t-cuda_35.4.1-20230801124926_arm64.deb**
@linuxdev is the log information helpful? please let me know how I might be able to proceed in recovery.
Hey, I was able to flash the Orin from the sdk-manager and all is good now as far as I can tell:
The reason why I wasn’t able to connect before is because I wasn’t using the right USB port on the Orin. With the notion that the connection should be via usb-A - usb-C cable I mistakenly tried to connect to one of the usb-A ports on the right of the 40 pins, instead of the usb-C on the left. (in my case, the Orin’s placement is a little different, where “front” panel is “left” (to avoid having the cables in front). Kind of a typical rubic-cube confusion :) so that prevented me from seeing or considering usb-C port on the left of the 40 pins. Walla…
Thanks for all your great help
Glad it worked out. Somehow now though I’m inspired to design an actual working Rubic’s Cube chassis for the Jetson… 😂
(perhaps the cube solution could be the password)
Some trivia you might be interested in: A USB-A port can never be used as a device mode (devices plug in to a USB-A, the USB-A is a host). One end of USB is always a device, and the other is always a host. A type-C port seems to violate this, but it isn’t the case; type-C ports have dual wiring, so they can be both host or device, but that’s because one set of wires would be host only, and one set would be device only (and if device is at one end only, and host at the other end only, the paths can be combined to do things like go from 5 Gb/s for one path to 10 Gb/s for combined paths). When in recovery mode a Jetson has wiring for only one port, which tends to differ between Jetsons, but is usually at the same physical location (it seems to be more like a theme than a rule).