USB C Ports Jetson AGX Xavier stopped working

Hi linuxdev,

thanks a lot for this explanation. It really helps me to understand more details of the Xavier and the structure and function of device trees.
You were right: tegra194-p2888-0001-p2822-0000.dtb is used on my Xavier. It was located at /boot/dtb. I converted it in a .dts file (with the commands you suggested, thanks for that) and added usbcore.autosuspend=-1 in bootargs as shown in the code below:

chosen {
		bootargs = "root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 usbcore.autosuspend=-1  ";

Is this the right way to add the command?
In the next step I reconverted this .dts file back to a .dtb file.
To apply the changes, the next step would be to flash the kernel? (I am asking that exactly, because I really would like to understand your suggestions correctly)
Because this is the first time, that I have to flash a kernel, I plan to create a clone of the rootfs partition. I do not understand this process completely right now, therefore I will continue reading on this topic and then probably come back with some concrete questions.

Thanks again for your help and time!

Be careful to not use “/boot/dtb” content directly. I say this because in the past the DTB file was always added there, but as time went on over many years of releases, this became signed content in a partition. The “/boot” DTB content is still there due to the script copy still copying, but the particular file is usually not used.

The real running system can be examined via “dtc -I fs -O dts -o extracted.dts /proc/device-tree”, but even this is not completely reliable. The reason is that earlier bootloader stages may edit the original device tree before passing it on to a next bootloader stage. The actual file which would be a match is the one of the same name on the host PC during the flash.

If “tegra194-p2888-0001-p2822-0000.dtb” is the correct file (and there is a good chance this is what was signed and put into a partition…flashing and examining the log would be a better way to find out if you need certainty), then you could perform those steps on the host PC’s version of this file, and then put the replacement back in on the host PC prior to flashing. Maybe this is an exact match to the “/boot” version, but I cannot guarantee this.

Your actual sample modification is indeed what you would need, though I would only leave one space at the end of the string (and I’m not sure if that one space is needed). Two won’t hurt, one might not even matter, but I know that two spaces would be unnecessary in all cases.

Once you boot with this modification in place you should see this show up from “cat /proc/cmdline”.

Hi linuxdev,
thanks a lot for this explanation!
I tested the command dtc -I fs -O dts -o extracted.dts /proc/device-tree (output is here: dts_source.log (52.2 KB)). Do we see here the nodes of the device tree? Is it normal that there are so many warnings inside?

For the procedure of flashing I found some instructions, but in everyone a host machine is needed. So now my question would be, if it is theoretically possible to connect a Xavier with its USB A port to a computer? I tested it and it did not work, but also my cable could be unfitting for this.

Edit: I tested to connect Jetson Xavier via its USB C port in recovery mode and it was connected to the host computer. I had a look on the files saved by SDK Manager on the host, but I could not find the device tree file. But from your suggestions I assume that it should be there?

There is definitely an error there. I couldn’t tell you exactly why, but dtc had basically no valid output when extracting from “/proc/device-tree/”. If you cd to “/proc/device-tree/”, and look around with “ls”, then the structure should more or less match the original device tree used during flash (there could be some edits from earlier boot stages).

Note that warnings alone might not matter, but this is basically the entire file. Are you able to post “extracted.dts” directly after renaming it to “extracted.txt”? Or is this what the log file is?

The only cable useful for flashing from a host is the USB-C cable, and for actual flash steps this is only valid in recovery mode. Generally speaking this is necessary for adding a new device tree (in part because this now goes into a partition, and also because the partition must be signed prior to it going in). The other models of Jetsons tend to use the micro-B USB, but for Xavier use only the USB-C (a type-A host connector is incapable of going into device mode…type-C has extra wiring, and the micro-OTG port on Xavier is not wired for flash…other Jetson models do use the micro-B).

Note that unless you’ve flashed once with JetPack/SDK Manager on your host, or manually unpacked the driver package and sample rootfs, that some of the host side files may appear to be missing although in reality this is not necessarily an error (e.g., some files are copied from reference locations into actual location of use during a flash…reference copies would still be there, but prior to first flash some files may appear to be missing on host side).

Hi linuxdev,

thanks for explaining the USB possibilities to connect the Jetson Xavier to the host machine.
The host machine is actually the machine, on which I installed SDK Manager and flashed the jetson Xavier with.

The file I posted yesterday is the output I recieve in the terminal on Xavier after I entered
dtc -I fs -O dts -o extracted.dts /proc/device-tree
I converted it into a txt file, but by any reason I was just able to upload it as log…

I made my first attempts to backup the Jetson Xaviers image. (I try it with this tutorial) I was able to connect to the Xavier from my host computer with ssh. When I tried to copy the partition mmcblk0p1 with
dd if=/dev/mmcblk0p1 | ssh user@hostpc dd of=/directory/image.raw I recieved the error:
port 22: Invalid argument
Might this be a problem wirh SSHOpenServer? I have it installed on my Xavier. When I tried to configurate SSHOpenServer with
man sshd_config
I got an reply that no manual changes are allowed. Could it be possible, that my filesystem stays in read-only mode after I once set it with
echo u > /proc/sysrq-trigger
in the first steps for the backup with the following code?

Many thanks again for your continouus replies!

What that URL does is to put the rootfs image into read-only mode prior to using dd to copy the entire partition (which is valid and should work). The ssh side is just piping the data from the dd read operation to the dd write operation on a remote system. I have not tried that specific command, but some comments may help. There may be cases where you need to adjust with “sudo”. I personally use private keys for ssh from my PC to Jetsons, and thus no password is used. Additionally, I temporarily activated the root account, added ssh keys for that, and then deactivated the root account again…as a result I can simply “ssh root@xavier_address” and it works without sudo (making ssh development dramatically easier while keeping the root account deactivated). I would actually have to remove my keys to experiment with using sudo directly, so YMMV.

If your Xavier has address “xv” (I’m abbreviating, maybe “xv” is really, and the host PC has the address “pc” (also an abbreviation, maybe “pc” is, then from the PC side you could use the PC to see and read various commands. For example the host could see the partition sizes with:
ssh user@xv df -H
…and you would see the “df -H” output.

The host could tell dd to run and combine that with the dd line on your PC, and is a rearrangement of what I see you used:
ssh user@xv "dd if=/dev/mmcblk0p1 | dd of=image.raw

I think your earlier command may have been intended as running from the Jetson side (instead of the PC side originating), and this could simplify sudo. A rearrangement which might work (haven’t tested) is something like this:
sudo dd if=/dev/mmcblk0p1 | ssh user@pc dd of=image.raw

Keep in mind that regardless of which side originates the dd command the Jetson side must be in read-only mode to prevent your image from changing in the middle of the stream (which would give you a corrupted copy of the actual partition). This part of your URL is the part where this is accomplished:
sudo echo u > /proc/sysrq-trigger

From the PC side you could try this if you do not want to be logged in to the Jetson:
ssh user@xv sudo echo u > /proc/sysrq-trigger

Keep in mind that the image which is cloned will be the image from a running system with your user logged in. An actual clone using and a recovery mode Jetson guarantees that not only is the partition read-only, but also that the state of the partition is without logins. If you had previously logged out and shut down correctly, then your image will reflect this. If you had previously performed an unclean shutdown, then the clone would also reflect this and try to replay the journal upon boot.

Since I cannot directly test, I’ll suggest what may have caused this:

Directly piping content to ssh may have interpreted a “-” in the data as an argument to dd or to ssh. This is where programs like cpio come in handy…they serialize this and essentially stop stray special characters from randomly being interpreted.

If I were to use cpio to edit an initrd (which is a cpio/tar archive filesystem), then it would go something like this (demonstrates unpacking an old initrd and then creating a new initrd from the edited content with cpio in the middle):

For your case take a close look at inserting this in the middle of your pipe prior to going to ssh:
sudo cpio --quiet -H newc -o

Take a close look at adding this to the final write of content at the other side of the ssh:
sudo cpio -idm

These are just inserted in the correct place with piple “|” symbols.

There are other ways to make a safe stream, e.g., rsync works with binary files and streams across two computers.

The simplest way is to just clone with Example if host PC has a lot of spare space (over 35GB) and the Jetson is attached in recovery mode:

sudo ./ -r -k APP -G my_backup.img jetson-xavier mmcblk0p1

Clones produce both a “.img” file and a “.img.raw” file. The sparse “.img” file is much smaller than the full raw clone, but is mostly worthless since it can only be used for flashing and there is no ability to manipulate or view the content. I throw out the “.img” and keep the “.img.raw”. If I am done using this then I run “bzip2 -9” on it for storage. Due to size even a simple file copy takes a long time, and bzip2 could take a couple of hours.

The raw clone can be loopback mounted, repaired, examined, so on, and then used to directly flash from it. In theory so could the dd version of this, but I would expect more consistent success under corner cases from clone.

Hi linuxdev,
I tried your suggestion of the command sudo ./ -r -k APP -G my_backup.img jetson-xavier mmcblk0p1 and it seems that worked. Thanks for that!
This command was executed in the directory, where is located. In my case it was SDKManager/SDKNvidia/JetPack_4.2.3_Linux_GA_P2888/Linux_for_Tegra (Just for the record and in the case, somebody might have the same issue).
Now I recieved a .raw and a .img file (as you described). I am just a bit wondering, that they are almost equally sized (.img is 29 GB, .raw is 30 GB) Is there any possibility to check, if the copy is done correctly?
From what I read I assume the .raw file contains a copy of the file system? Does that mean, if I flash with this file, the complete directory structure (including for example /home) is recreated?
When I had a look on the Jetson Xavier, there were some more partitions called dev/mmcblk0p*. Is it right, that they can be ignored for doing a clone?
I also found on my host computer the tegra194-p2888-0001-p2822-0000.dtb (as well located in /SDKManager/SDKNvidia/JetPack_4.2.3_Linux_GA_P2888/Linux_for_Tegra/kernel/dtb). After converting to .dts I changed the node chosen from:

chosen {
	bootargs = "console=ttyTCU0,115200";


chosen {
	bootargs = "console=ttyTCU0,115200, usbcore.autosuspend=-1";

So now, after reconverting it to .dtb I should do the kernel flash (e.g. this)?
Thanks for your detailed explanation and the suggestion for storing!

An explanation of what a raw and sparse files are may help here. A raw file is literally all of the bytes of the full partition as purely binary data (there is no awareness of ext4 filesystem formatting). A sparse file understands the filesystem type ext4, and does not treat this as purely binary data.

In the sparse file where ext4 is understood the total size of the partition is known, and then blocks of empty parts of the filesystem are discarded…along with metadata about the discarded empty blocks. “Sparse” files are (at risk of being inexact) something like a run length compression when a series of empty spots just have a number, and during flash that number of empty bytes are created by algorithm instead of being stored in memory.

The sparse file, if from a filesystem which is 100% full, will match the raw file. The emptier the filesystem, the smaller the sparse file gets. The raw file will always be an exact byte size no matter what its content is.

If you were to divide the exact byte size of the raw image by 1024, then this is the size in KiB. If you divide twice by 1024, then this is the size in MiB. If you divide three times by 1024, then this is GiB. The flash software accepts sizes only in MiB or GiB increments. If you divide twice by 1024 and this is a non-integer fraction, then the clone was in error. If you can divide the size by 10234 three times, then this too is a valid size. No other sizes are valid so far as the script is concerned. The sparse file has no method to validate it.

There is open source software to work with sparse images, but these seem to differ from the method NVIDIA uses, and so I’ve never found any tools which actually have the ability to turn the sparse format of the NVIDIA image into a raw format. The “bootloader/mksparse” program can convert a raw file into a sparse file, but apparently not the reverse.

Note that if you use a clone to restore from, and if the clone does not match the default flash image size, then it might be useful to use the “-S size” parameter to to specify the non-default size. Taking your 28GiB as an example:
sudo ./ -S 28GiB jetson-xavier mmcblk0p1

In a case where your filesystem is so full there wouldn’t be much of a speed advantage by flashing a sparse image. A nearly empty filesystem gets a much faster flash advantage if the sparse image is used (relative to the raw image flash time).

Does your image divide evenly by 1024 at least twice? Then it is valid. I recommend throwing away the sparse image and keeping the raw image.

Your entire rootfs is a 100% match of the clone. If you have not created some custom layout where “/home” is on its own partition, then 100% of everything is there, guaranteed.

Keep in mind that dynamically generated content, e.g., some of the device special files in “/dev”, or “/proc” or “/sys”, are not part of the partition.

You can do this to examine what is there and convince yourself:

cd /mnt/home
df -H -T .
sudo umount /mnt

The other non-rootfs partitions are used for boot. These include a lot of operations which people would not normally customized, for example, operations which a PC might consider BIOS setup. End users here may customize this non-rootfs content when working working with secure boot or making custom drivers which must be active prior to reaching the Linux kernel boot stage. These other partitions can normally be wiped out and restored without risk for default configurations.

The clone is just the rootfs, but just the rootfs is normally all you’d need to worry about during a restore.

For the topic of the device tree some of the details change depending on release. Mostly the way to do this would be to take your modified “.dtb” file and overwrite the PC host’s version of this in its location somewhere in either “Linux_for_Tegra/bootloader/” or “Linux_for_Tegra/kernel/”. Then do a normal command line flash (well, you would put a copy of your “my_backup.img.raw” as “Linux_for_Tegra/bootloader/system.img”, and add the option to “reuse” the image). Something like this if your image is 28GiB:

cd /where/ever/it/is/Linux_for_Tegra/bootloader
cp /where/ever/it/is/my_backup.img.raw /where/ever/it/is/Linux_for_Tegra/bootloader/system.img
cd /where/ever/it/is/Linux_for_Tegra/...either bootloader subdir or kernel subdir.../
cp /where/ever/it/is/custom_dtb.dtb /where/ever/it/is/Linux_for_Tegra/...kernel or bootloader original file...

IMPORTANT: Be careful to save a backup of any device tree you overwrite. You could simply unpack a new one later if you lose one.

Then, from the “Linux_for_Tegra/” location:

sudo ./ -S 28GiB -r jetson-xavier mmcblk0p1

If you want to log this while you flash, and use the log to verify the correct dtb was used, then you could add this:

sudo ./ -S 28GiB -r jetson-xavier mmcblk0p1 2>&1 | gawk '{gsub("[0-9][0-9]+[/][0-9[0-9]+ bytes sent..",".");print}' | tee log.txt

(the gawk reduces progress bar spam)

if you connect usb-c port J512 of Xavier to host PC
then run
lsusb at the latter
will the Host PC see the Nvidia Corp device?

Press Power button and release
Press Force Recovery button; keep pressing it
Press Reset button and release
Release Force Recovery button
$ lsusb

@Andrey1984 Yes, this works.


it seems that you managed to create a raw disc image;
you may like to mount it as a loopback e.g at Host PC:

  1. assign your image to a loop device:
sudo losetup -f $HOME/path/to/disk.img
  1. find the path to that it mounted
sudo losetup -j $HOME/path/to/disk.img
/dev/loop0: [xxxx]: (/home/path/to/disk.img)
3. mount the device to folder
sudo mount /dev/loop0p1 /mnt

However, it seems that you should be able to flash the jetson with sdkmanager if that will be required.

Hey @Andrey1984,
thanks for your suggestion.
Yes, I assume flashing with sdk manager should work.
I do not completely understand what is the advantage of mounting the image (Do you mean .img or .raw file in your upper post?). Is it done instead of flashing?
@linuxdev I am reading your post right now, will take some time…

if you mount the image to the Host PC you will be able to access files from it;
img in my example is a placeholder and it means that any raw image file might be placed in its place;
Probably reflashing the device might resolve the original issue.

Hi @linuxdev,

thank for all this explanations. Especially the calculation for the images functionality and the theory behind .raw files were very helpful.
The size of my .raw file can be divided by two times 1024 to an int, so I assume the image should work.

I replaced the custom.dtb file in the directory Linux_For_Tegra/kernel/dtb and also in Linux_For_Tegra/bootloader, because I found the original file on both places.
Before flashing, I got one question to this line:

Do I have to replace system.img with my custom.raw.img or do I have to replace system.img.raw? In both cases, do I have to remove the other “old” file (system.img or system.img.raw)?

Btw. on the Xavier the /mnt directory was empty…

Again I really appreciate your explanations. Thanks for that.

@Andrey1984, also thank you too for your last post.

Normally “bootloader/system.img” is a temporary file. A huge temporary file. Flash without the “-r” option will regenerate this. Actually, normal flash regenerates “bootloader/system.img.raw” every flash, and then creates a sparse version as “bootloader/system.img”. “system.img.raw” could be renamed as “system.img”, and the result would be exactly the same flash, but take longer to flash.

You need to place your clone raw image in place of “bootloader/system.img”, and keep your clone as name “system.img” without regenerating the file during the flash (without “-r” you won’t be flashing your clone image…a new image would overwrite what you put there…do not forget “-r”).

You can always delete “bootloader/system.img.raw” to save space if you want, but it is there for you to examine after a flash if you want to explore it. Other than being used to create a default sample rootfs system.img this file has no purpose other than your curiosity. I would delete “bootloader/system.img.raw” and save space.

Incidentally, the “mksparse” application can be used to create a sparse version of your raw image. In your case where the filesystem is nearly full this won’t save much flash time and will cost about 28GB of disk space on the host PC. A sparse file in a production environment saving only a few seconds could help, but is not worth much other than costing effort when running a single flash and not repeated hundreds of times per day. The default sample sparse image is so much smaller than the full filesystem that it makes sense to convert to sparse before flash even if it is just for one flash.

The “/mnt” directory is just an example of where to mount. “/mnt” is more tradition than anything else. If a disk or loopback image is mounted there, then you should see any content there. Just make sure mount actually succeeded. You can also run command “df -H -T /mnt” to see what is mounted there (before and after mount) and verify the loopback image is mounted versus the actual eMMC or hard disk. If the loopback image is empty, then it probably was not a valid clone (after a successful “mount -o loop imagename /mnt” there should be new content in “/mnt” until the “umount”, and “df -H -T /mnt” should change depending on mount or not mount).

On the Xavier and even on a host PC expect “/mnt” to be empty unless you’ve modified something or mounted something there. If something is already there, and you mount there, then the modified content will temporarily become inaccessible and be replaced with the mount content. Once mount is removed the old content will once again become accessible. I take advantage of this on my host PC by copying my “/home” content into the actual home directory, and then mounting my home content over the top of it with a RAID partition. If my RAID partition fails, or if I umount it, then I still have home directory content (but out of date). Useful for maintenance.

Individual users have an equivalent to “/mnt”: “/var/run/user/<numeric_user_id>”. Think of “/mnt” as the manual mount location for user root to customize, and “/var/run/user/” as the auto mount location for any user.

Hi linuxdev,

thanks as always for the detailed information.
I flashed my Jetson Xavier on my host computer with running the command

sudo ./ -S 28GiB -r jetson-xavier mmcblk0p1 2>&1 | gawk '{gsub("[0-9][0-9]+[/][0-9[0-9]+ bytes sent..",".");print}' | tee log.txt

I uploaded the logs that were created (as .log file, even if they were .txt files) logFlash.log (1.9 MB) logFlash3.log (1.9 MB) logFlash2.log (1.9 MB). It seems, that it flashed successfully.

After this command was executed, I pressed Jetson Xaviers reset button to start it. Everything went normal, files were there as always. Unfortunately I still receive the same output for cat /proc/cmdline:

root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa0697000 lut_mem=0x2008@0xa0693000 usbcore.old_scheme_first=1 tegraid= maxcpus=8 boot.slot_suffix= boot.ratchetvalues=0.4.2 vpr=0x8000000@0xf0000000 sdhci_tegra.en_boot_part_access=1

Also /sys/module/usbcore/parameters/autosuspend still gives the output 2.

I did two things for flashing:

  1. cp my_Backup.img.raw /SDKManager/SDKNvidia/JetPack_4.2.3_Linux_GA_P2888/Linux_for_Tegra/bootloader/system.img 

I deleted the old system.img file and the system.img.raw file.

  1. I added usbcore.autosuspend=-1 in the chosen node in the files Linux_for_Tegra/kernel/dtb/tegra194-p2888-0001-p2822-0000.dts and Linux_for_Tegra/kernel/dtb/tegra194-p2888-0001-p2822-0000.dtb.
    They had different entries inside the chosen-node.

The chosen node in the .dtb-file in /kernel/dtb looks like this:

chosen {
	bootargs = "console=ttyTCU0,115200, usbcore.autosuspend=-1";

The chosen node in the .dtb-file in /bootloader looks like this:

chosen {
	bootargs = "root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 usbcore.autosuspend=-1    ";

When the flashing is done, and I look beck into the converted .dts file, the part usbcore.autosuspend=-1 is removed. (I double checked multiple times). In total I repeated the flash three times, to which the log-files belong.
I also tried to the usbcore parameter in extlinux.conf and also in usbcore.conf.
Am I missing something, or why does the flashing not result in what we expected?

Looks like my gawk filter is leaving in the progress bar (sadly, this makes the log size huge just from the progress bar).

The flashes were correct and successful. Which exact file name and location did you place this in on your host’s “Linux_for_Tegra/” content? If the wrong dtb file were copied in, then it might miss the updates to “chosen->bootargs”. The log file should be able to verify if your file was actually copied in during the flash.

Hi linuxdev,

the file, in which I added the option usbcore.autosuspend = 1 is
Its reconverted .dtb file is located in /SDKManager/SDKNvidia/JetPack_4.2.3_Linux_GA_P2888/Linux_for_Tegra/kernel/dtb/.
In the log, lin 137, the file is mentioned the first time.

Just for the check, after flashing the Jetson Xavier I reconverted tegra194-p2888-0001-p2822-0000.dtb to .dts. The option usbcore.autosuspend=-1 still was written there.

Also I added the usbcore-option to another file named alsotegra194-p2888-0001-p2822-0000, located in /SDKManager/SDKNvidia/JetPack_4.2.3_Linux_GA_P2888/Linux_for_Tegra/bootloader. This did not seem to have an impact, because there the option was removed after flashing.

I also would like to add two other things, that I tried and found and that might be helpful for finding a solution.

There is a difference in the dmesg | grep -i usb output depending on which USB C port is connected. I tested it by connecting one of the USB C ports to my host PC while being logged in in Jetson Xavier.
If I connect J512 it gives:

[ 46.911465] l4tbr0: port 2(usb0) entered blocking state
[ 46.911473] l4tbr0: port 2(usb0) entered forwarding state
whole dmesg | grep -i usb here: dmesgGrep-iUSB_Output_J512_front.log (6.4 KB)

If I use J513 the output usb usb2: usb_suspend_both: status -16 (as shown in my first post) is generated.
Also sometimes there is a log different from the first one, giving these errors/warnings:
[ 13.416769] usb 1-4-port1: Cannot enable. Maybe the USB cable is bad?
[ 3918.126020] usb usb2: usb_suspend_both: status -16
[ 3918.126457] usb usb2-port1: config error
[ 3926.269932] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
usb usb2: usb_suspend_both: status 0

I did not find a logic behind, if one of the different outputs in Port J513 is generated.

The second thing I found is, that it was temporarily possible to enable usbcore.autosuspend=-1 (Found here)
In the file sys/module/usbcore/parameters/autosuspend I could set the number from 2 to -1. The output $ cat /sys/module/usbcore/parameters/autosuspend was -1, but with no effect on the functionality of the USB C ports.
I also tried the command
sudo sh -c "echo -1 > /sys/bus/usb/devices/2-0:1.0/power/autosuspend_delay_ms"
but even as being logged in as root I had no permissions to execute this command.

If you suspect the hardware may be broken, please try to re-flash the whole system through SDKManager and check if the two ports work. If it is still does not work, you may consider RMA.

Please check

The “/sys” files are not real files, but are drivers printing data about themselves, and only sometimes it will be possible to actually write (it depends on the source code of the driver). You may find places where these files are a reflection of current settings, but not usable as an actual method to impose changes.

As mentioned by @DaneLLL it is probably time to consider reflashing from scratch to see if this has been some other issue in software, versus some sort of failure in need of RMA.