USB errors using external drive, works fine on host

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

I’m trying to use an external NVMe drive in a USB adapter. It works fine on my Ubuntu host and Mac, but I get a lot of errors reading and writing it on the Jetson Nano, ultimately resulting in the drive being unmounted and re-mounted read-only:

[67435.685328] scsi 0:0:0:0: Direct-Access     JMicron  Generic          0208 PQ: 0 ANSI: 6
[67437.709964] sd 0:0:0:0: [sda] 1000215216 512-byte logical blocks: (512 GB/477 GiB)
[67437.717620] sd 0:0:0:0: [sda] 4096-byte physical blocks
[67437.724014] sd 0:0:0:0: [sda] Write Protect is off
[67437.729242] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[67437.738863] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes not a multiple of physical block size (4096 bytes)
[67437.758162] sd 0:0:0:0: [sda] Attached SCSI disk
[67510.999956] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[67686.568678] scsi 0:0:0:0: Direct-Access     JMicron  Generic          0208 PQ: 0 ANSI: 6
[67688.597497] sd 0:0:0:0: [sda] 1000215216 512-byte logical blocks: (512 GB/477 GiB)
[67688.605104] sd 0:0:0:0: [sda] 4096-byte physical blocks
[67688.611620] sd 0:0:0:0: [sda] Write Protect is off
[67688.616866] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[67688.626483] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes not a multiple of physical block size (4096 bytes)
[67688.646311] sd 0:0:0:0: [sda] Attached SCSI disk
[67692.539007] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67692.548563] tegra-xusb 70090000.xusb: @00000000ffebcb70 ffe14000 00000000 04000000 05088000
[67758.229462] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67758.239017] tegra-xusb 70090000.xusb: @00000000ffebc850 ffe14800 00000000 04000000 05088001
[67896.568263] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67896.577819] tegra-xusb 70090000.xusb: @00000000ffebc9b0 ffe14800 00000000 04000000 05088000
[67921.401526] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67921.411082] tegra-xusb 70090000.xusb: @00000000ffebcde0 ffe14800 00000000 04000000 05088001
[67925.077696] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67925.087249] tegra-xusb 70090000.xusb: @00000000ffebca90 ffe14000 00000000 0a010000 05088000
[67928.085740] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67928.095293] tegra-xusb 70090000.xusb: @00000000ffebc1d0 ffe14800 00000000 0a000060 05088001
[67930.101719] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67930.111272] tegra-xusb 70090000.xusb: @00000000ffebc950 ffe14000 00000000 0a000060 05088001
[67932.226780] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67932.236336] tegra-xusb 70090000.xusb: @00000000ffebc860 ffe14000 00000000 04000000 05088000
[67934.216658] scsi_io_completion: 14 callbacks suppressed
[67934.222112] blk_update_request: 14 callbacks suppressed
[67934.222114] blk_update_request: I/O error, dev sda, sector 495616
[67934.233530] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 109051904 size 8388608 starting block 62208)
[67934.247291] Buffer I/O error on device sda1, logical block 61696
[67934.253344] Buffer I/O error on device sda1, logical block 61697
[67934.259464] Buffer I/O error on device sda1, logical block 61698
[67934.265700] Buffer I/O error on device sda1, logical block 61699
[67934.271748] Buffer I/O error on device sda1, logical block 61700
[67934.277872] Buffer I/O error on device sda1, logical block 61701
[67934.283889] Buffer I/O error on device sda1, logical block 61702
[67934.290005] Buffer I/O error on device sda1, logical block 61703
[67934.296022] Buffer I/O error on device sda1, logical block 61704
[67934.302033] Buffer I/O error on device sda1, logical block 61705
[67934.309137] blk_update_request: I/O error, dev sda, sector 494592
[67934.315265] blk_update_request: I/O error, dev sda, sector 493568
[67934.321398] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 109051904 size 8388608 starting block 61952)
[67934.335777] blk_update_request: I/O error, dev sda, sector 492544
[67934.342786] blk_update_request: I/O error, dev sda, sector 491520
[67934.348911] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 100663296 size 8388608 starting block 61696)
[67934.363058] blk_update_request: I/O error, dev sda, sector 490496
[67934.369192] blk_update_request: I/O error, dev sda, sector 489472
[67934.375454] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 100663296 size 8388608 starting block 61440)
[67934.389738] blk_update_request: I/O error, dev sda, sector 488448
[67934.395867] blk_update_request: I/O error, dev sda, sector 487424
[67934.401974] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 100663296 size 8388608 starting block 61184)
[67934.416167] blk_update_request: I/O error, dev sda, sector 486400
[67934.423217] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 100663296 size 8388608 starting block 60928)
[67934.438116] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 100663296 size 8388608 starting block 60672)
[67934.452466] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 100663296 size 8388608 starting block 60416)
[67934.467303] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 100663296 size 8388608 starting block 60160)
[67934.481417] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 100663296 size 8388608 starting block 59904)
[67936.222207] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67936.231764] tegra-xusb 70090000.xusb: @00000000ffebc390 ffe14000 00000000 04000000 05088001
[67940.062010] scsi_io_completion: 14 callbacks suppressed
[67940.067313] blk_update_request: 14 callbacks suppressed
[67940.067316] blk_update_request: I/O error, dev sda, sector 505856
[67940.106635] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67940.116193] tegra-xusb 70090000.xusb: @00000000ffebc670 ffe14800 00000000 04000000 05088001
[67944.198600] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67944.208159] tegra-xusb 70090000.xusb: @00000000ffebcfb0 ffe14800 00000000 04000000 05088001
[67948.164075] blk_update_request: I/O error, dev sda, sector 506880
[67948.170541] EXT4-fs warning: 2 callbacks suppressed
[67948.170546] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 109051904 size 8388608 starting block 63488)
[67948.189168] buffer_io_error: 3013 callbacks suppressed
[67948.189172] Buffer I/O error on device sda1, logical block 62976
[67948.205443] Buffer I/O error on device sda1, logical block 62977
[67948.211501] Buffer I/O error on device sda1, logical block 62978
[67948.217533] Buffer I/O error on device sda1, logical block 62979
[67948.223561] Buffer I/O error on device sda1, logical block 62980
[67948.229668] Buffer I/O error on device sda1, logical block 62981
[67948.235697] Buffer I/O error on device sda1, logical block 62982
[67948.241732] Buffer I/O error on device sda1, logical block 62983
[67948.247756] Buffer I/O error on device sda1, logical block 62984
[67948.253786] Buffer I/O error on device sda1, logical block 62985
[67948.278544] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67948.288098] tegra-xusb 70090000.xusb: @00000000ffebc970 ffe14800 00000000 04000000 05088000
[67950.156154] blk_update_request: I/O error, dev sda, sector 508928
[67950.162333] blk_update_request: I/O error, dev sda, sector 507904
[67950.168455] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 109051904 size 8388608 starting block 63744)
[67952.174471] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67952.184028] tegra-xusb 70090000.xusb: @00000000ffebc110 ffe14800 00000000 04000000 05088001
[67954.147848] blk_update_request: I/O error, dev sda, sector 515072
[67954.154186] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 13500445 (offset 117440512 size 8388608 starting block 64512)
[67954.168036] buffer_io_error: 502 callbacks suppressed
[67954.168052] Buffer I/O error on device sda1, logical block 64000
[67954.179433] Buffer I/O error on device sda1, logical block 64001
[67954.186159] Buffer I/O error on device sda1, logical block 64002
[67954.192301] Buffer I/O error on device sda1, logical block 64003
[67954.198671] Buffer I/O error on device sda1, logical block 64004
[67954.204776] Buffer I/O error on device sda1, logical block 64005
[67954.211172] Buffer I/O error on device sda1, logical block 64006
[67954.217456] Buffer I/O error on device sda1, logical block 64007
[67954.223554] Buffer I/O error on device sda1, logical block 64008
[67954.229638] Buffer I/O error on device sda1, logical block 64009
[67954.236041] blk_update_request: I/O error, dev sda, sector 29960
[67954.242590] blk_update_request: I/O error, dev sda, sector 28936
[67956.408247] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67956.417803] tegra-xusb 70090000.xusb: @00000000ffebcdd0 ffe14000 00000000 04000000 05088000
[67962.174348] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67962.183902] tegra-xusb 70090000.xusb: @00000000ffebc940 ffe14800 00000000 04000000 05088000
[67966.188299] JBD2: Detected IO errors while flushing file data on sda1-8
[67966.194960] Aborting journal on device sda1-8.
[67966.209425] EXT4-fs error (device sda1) in ext4_do_update_inode:5088: Journal has aborted
[67966.301146] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67966.310702] tegra-xusb 70090000.xusb: @00000000ffebc940 ffe14000 00000000 04000000 05088001
[67970.101028] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67970.110581] tegra-xusb 70090000.xusb: @00000000ffebc390 ffe14800 00000000 0a010000 05088000
[67972.248472] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal
[67972.248477] EXT4-fs error (device sda1): mpage_map_and_submit_extent:2515: comm kworker/u8:3: Failed to mark inode 13500445 dirty
[67972.259681] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67972.259686] tegra-xusb 70090000.xusb: @00000000ffebcb00 ffe14800 00000000 04000000 05088001
[67972.286779] EXT4-fs (sda1): Remounting filesystem read-only
[67978.100931] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67978.110484] tegra-xusb 70090000.xusb: @00000000ffebc170 ffe14800 00000000 0a010000 05088001
[67982.222095] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[67982.231649] tegra-xusb 70090000.xusb: @00000000ffebcf80 ffe14800 00000000 04000000 05088001
[67987.062396] EXT4-fs error (device sda1) in ext4_writepages:2825: IO failure
[67987.070810] JBD2: Detected IO errors while flushing file data on sda1-8
[68048.077923] sd 0:0:0:0: [sda] Synchronizing SCSI cache

I have a 180-13448-DAAA-A02 module and 180-13449-DAAF-A02 carrier board.

3

Hi rmann,

Is that the devkit carrier board?

It seems it is still an USB storage.

Please provide the full flash log for further check.

4

Yes, it’s the dev kit carrier.

Where do I get the full flash log?

5

You should connect the USB drive on the devkit.
And do the steps of your another post to flash into USB drive.
When you flash the board, the flash log would show on your console of host PC.
Just provide them as file here for further check.

6

Given that I couldn’t figure out how to flash a filesystem on the drive, I decided to just use it as external storage mounted on the Jetson Nano. Not ideal, but solves most of my issues. It was during checkout of a large codebase that I ran into those USB errors, and then the checkout ultimately failed. There’s something wrong with the USB on my Jetson. It’s running 18.04 (the current SD Card image 461).

7

Are you saying I can flash the drive from the Jetson installation, and I don’t need a host machine? The steps I linked above say to use the host machine.

In any case, the USB errors are preventing me from doing anything.

8

What Kevin tries to say is we want to check under what scenario will this usb drive causes error.
And we need the kernel log to tell what happenes to your usb.

The method to dump full kernel log is command “dmesg”

As for what you are asking:

Are you saying I can flash the drive from the Jetson installation, and I don’t need a host machine?

What we said previously is you can manually “prepare” a usb drive which jetson can boot from. But that still needs the help of a ubuntu host because the prepare work requires to use the BSP (downloaded by sdkm) too.

Also, just my personal suggestion, why not just use a native USB SSD instead of a adapter?

9

What I pasted above is what is output to the serial console when I am cloning the repo.

The full dmesg output, as currently available, is here. It has already wrapped and elided the lines since boot.

As for what you are asking:

Are you saying I can flash the drive from the Jetson installation, and I don’t need a host machine?

That’s what I thought, but it sounded like he was telling me to prepare the drive on the Jetson itself. I don’t have an 18.04 x86 machine to host on.

If I had a large enough one, I would. But I don’t. I bought this NVMe hoping to use it in the $200 carrier board I bought from Seeed Studio, but my module is -A02, and won’t work with it. I happen to have a USB<->NVMe adapter, so I’m using that. The adapter and the drive work fine on my 22.04 x86 machine (and my Mac). They don’t seem to work reliably on the Jetson, as evidenced by all the tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring log lines.

Maybe the module and dev kit are just too old. I can see the rework done on the carrier at the USB ports.

If Arrow Electronics overcomes their extraordinary incompetence, I should have an Orin Nano dev kit here on Thursday.

Any chance JetPack 5 will ever support 22.04?

10

It will be ubuntu 22.04 in future for Orin. But it will directly be jetapck6 at that time.

BTW, could you attach your dmesg as a file here? NV network bans the pastebin website for a long time. Thanks.

11

log.txt (91.2 KB)

12

So long as JetPack 6 supports the Orin Nano dev kit I just ordered, that’s great news. Is that coming soon?

13

Regarding the log, does this issue require this long time to reproduce?

I mean your error log directly starts in 67932 sec which means the device already booted up for almost one day.

If you reboot the device now, will it also hit issue? Also, does this issue get happened right after you set up the drive? or it really took a while to hit?

Just a FYI, such issue requires debug so we need to clarify the scenario in very detail. And we may need to get same device as yours and reproduce on our side. Thus, it will not have immediate solution.

As for your question about Orin Nano ubuntu 22.04, unfortunately, it will not be ready soon. What is the purpose to must use “ubuntu 22.04” ? I mean is there any specific feature that only work with 22.04?

14

I don’t believe it needs to be up and running for any length of time for that to occur. I just need to hit the drive hard. For example, I initially tried formatting the drive on the Jetson nano, and it failed (with very similar USB issues like you see in the log). At first I thought it was the drive, so I took it over to the 22.04 machine to format it there, and it worked fine.

Then I took it back to the Jetson Nano, and started cloning the Swift toolchain (which I have to build from source because there’s no apt package for recent Swift for 18.04). The failures happened again. so I put the drive back in the 22.04 machine to clone the files, which worked fine, and that’s when I started this topic.

The machine already has 22.04 on it. I use it for other things, and don’t want to downgrade it. I’m not even sure the motherboard is supported as well in 20.04. 22.04 has been out for a year. It has more recent packages available, with all the attendant features and security fixes.

I’ll be able to use 20.04 on the Orin Nano much better than I will 18.04, but this excessive coupling between host and target versions is incredibly cumbersome. Your engineers really need to reconsider their approach to the design of this software platform.

15

I just tried this. No, I get the errors as soon as I try to hit the disk hard. On a hunch, I decided to try a different USB port. For a while I thought this port was doing better, but it’s still getting those errors:

[  788.154526] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[  788.164083] tegra-xusb 70090000.xusb: @00000000ffebc7b0 ffe24800 00000000 04000000 05088001

So far they haven’t caused an error higher up in the filesystem.

16

Could you share the log so far? I know you only hit the error in timestamp 788, but we need full dmesg to tell the kenrel status and other usb driver log too.

17

dmesg.txt (86.1 KB)

Where is the other usb driver log?

18

The other port seems to be better, but I’m compiling the toolchain now, which is probably not hitting the drive as hard as cloning the repo or formatting it did. Oh, but it did get errors when I tried compiling on the other port.

Update: Eventually the USB errors got to be so numerous that the filesystem failed again:

console.txt (45.9 KB)

19

First, I think you must get the machine back into your possession before we can suggest and test any possible solutions.

20

Some good news: I picked up a Samsung SSD, and it’s working much more reliably. Some interaction between the Nano and the USB-NVMe I was using made it impossible. Hopefully the NVMe itself is good to use on the Orin Nano dev kit that should arrive tomorrow.

22

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.