Jetson TX-2 fail to flash; Ubuntu 16; Jetpack 3.1, 3.2

gopack · February 7, 2018, 12:56pm

Can’t flash the Jetson TX2 from Ubuntu 16(not VM) using both Jetpack 3.1 and 3.2.
Using original usb cable.
Device recognized on lsusb as

Bus 001 Device 018: ID 0955:7c18 NVidia Corp.

Or on lsusb -t:

Port 4: Dev 18, If 0, Class=Vendor Specific Class, Driver=, 480M

And lsusb -v:

Bus 001 Device 018: ID 0955:7c18 NVidia Corp. 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0955 NVidia Corp.
  idProduct          0x7c18 
  bcdDevice            0.00
  iManufacturer           1 NVIDIA Corp.
  iProduct                2 APX
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           32
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xc0
      Self Powered
    MaxPower               32mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass    255 Vendor Specific Subclass
      bInterfaceProtocol    255 Vendor Specific Protocol
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass          255 Vendor Specific Class
  bDeviceSubClass       255 Vendor Specific Subclass
  bDeviceProtocol       255 Vendor Specific Protocol
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0001
  Self Powered

Bus 001 Device 017: ID 24ae:1000  
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)

Got this output from 3.1

./tegraflash.py --chip 0x18 --applet "/home/user/Downloads/jetpak/64_TX2/Linux_for_Tegra_tx2/bootloader/mb1_recovery_prod.bin" --cmd "dump eeprom boardinfo cvm.bin" --skipuid 
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0017 ] Generating RCM messages
[   0.0027 ] tegrarcm_v2 --listrcm rcm_list.xml --chip 0x18 --download rcm /home/user/Downloads/jetpak/64_TX2/Linux_for_Tegra_tx2/bootloader/mb1_recovery_prod.bin 0 0
[   0.0036 ] RCM 0 is saved as rcm_0.rcm
[   0.0041 ] RCM 1 is saved as rcm_1.rcm
[   0.0041 ] List of rcm files are saved in rcm_list.xml
[   0.0041 ] 
[   0.0041 ] Signing RCM messages
[   0.0051 ] tegrasign_v2 --key None --list rcm_list.xml --pubkeyhash pub_key.key
[   0.0062 ] Assuming zero filled SBK key
[   0.0100 ] 
[   0.0100 ] Copying signature to RCM mesages
[   0.0108 ] tegrarcm_v2 --chip 0x18 --updatesig rcm_list_signed.xml
[   0.0118 ] 
[   0.0119 ] Boot Rom communication
[   0.0129 ] tegrarcm_v2 --chip 0x18 --rcm rcm_list_signed.xml --skipuid
[   0.0138 ] Boot Rom communication failed
[   3.0132 ] 
Error: Return value 3
Command tegrarcm_v2 --chip 0x18 --rcm rcm_list_signed.xml --skipuid
Reading board information failed.

And this one from 3.2

###############################################################################
# L4T BSP Information:
# R28 (release), REVISION: 2.0, GCID: 10136452, BOARD: t186ref, EABI: aarch64, 
# DATE: Fri Dec  1 14:20:33 UTC 2017
###############################################################################
Error: probing the target board failed.
       Make sure the target board is connected through 
       micro-B USB port and is in recovery mode.

Please help.

linuxdev · February 7, 2018, 1:06pm

Is the cable you used the micro-B USB supplied with the kit? Was there any USB HUB in the middle?

gopack · February 7, 2018, 1:32pm

I tried both.
I used supplied micro-USB cable first.
And then I used it through usb-hub.

Results was same.

linuxdev · February 7, 2018, 5:46pm

I would actually advise deleting your entire JetPack directory and downloading it again. If that fails, then command line flash via driver package plus sample rootfs. If it still fails it may be a hardware problem.

gopack · February 8, 2018, 6:23am

I tried another PC and there was the same error. And I tried the flash via command line as well…
Got the same results.
So I should send my Jetson TX2 board to local (Russia, Moscow) support, so they will fix my hardware problem or give me another machine?

gopack · February 8, 2018, 7:29am

By the way board itself works, except mine x-server not working and I was not able to install it properly.

So I used to work in consoles (Ctrl+Alt+F[1-6])

linuxdev · February 8, 2018, 9:32pm

Recovery mode is unusually reliable and simple. If this fails, and if you have ruled out the host PC and USB cable (it must be the micro-B USB cable…chargers and other devices usually have a cheap or incorrect version), then there isn’t much else to try short of replacing the hardware. If two separate computers have the issue, and if the source of the flash software was not in common (both shouldn’t be from the same download), then probably the issue is not the host.

Having a system which has some working boot, yet has a failed recovery mode, is unusual. I can’t absolutely guarantee it is the Jetson which is bad, but since the next step would normally be exchanging the Jetson itself and trying again, I can’t offer a better solution. I would talk to the vendor and see if replacement can be done. Without flashing (i.e., without being able to upgrade to recent versions and never being able to reinstall if something goes wrong) the usefulness of the Jetson will be severely limited.

gopack · February 13, 2018, 7:41am

Thanks and hello.
I somehow successfully flashed the system through another PC in VM, Ubuntu 14.
Cant understand the particular reason of error which had occurred before.
Maybe it was because of Laptop and Ubuntu version (Dell Inspiron 15 5000 series, Ubuntu 16)

linuxdev · February 13, 2018, 3:08pm

Unless the Dell is not x86_64 it should work. Even the 64-bit Atom laptops should work (I have one sitting here I’ve used with JetPack3.1). VMs themselves are usually trouble.

Regardless of host it is odd that the Jetson itself did not want to enter recovery mode. This should be independent of host, e.g., even another Jetson would be able to see a recovery mode Jetson in lsusb (another Jetson just wouldn’t be able to make use of that). Regardless, I’m glad you got it to work without replacing.

tao.wang · July 12, 2018, 7:11pm

I got the same issue. I tried on two developer kit, two J130 (Auvidea); Jetpack 3.2, 3.2.1, 3.1. Tried with JetPack-L4T-3.2.1-linux-x64_b23.run and also flash.sh. All say cannot probe the device.

linuxdev · July 12, 2018, 7:15pm

Is this a VM? Are you using the micro-B USB cable which came with the dev kit (or a cable you have successfully flashed with before)?

tao.wang · July 12, 2018, 7:38pm

No, I am using a Dell Ubuntu 16.04 laptop XPS 13 9343. The cable I used is ok to upload firmware to other devices. I changed a cable, still the same.

tao.wang · July 12, 2018, 7:42pm

I am not sure if this is also should be mentioned: there is no /dev/mmcblk0p1 device can be seen.

linuxdev · July 12, 2018, 7:55pm

“/dev/mmcblk0” would be visible if the driver sees eMMC. The partitions themselves won’t be seen unless the drive is partitioned correctly, and it is probable that a failed flash would erase partitions. If the Jetson was a module instead of the dev kit, then there will be no partitioning at all and mmcblk0p1 would be missing (only the dev kit comes with software pre-installed…there won’t be any partitions with a bare module until it is successfully flashed).

Since you are using a third party carrier board I am thinking you purchased a bare module. Is this correct?

I’m not sure about what to check in the case of the alternate carrier board, but does the recovery mode Jetson show up on the host PC with this?

lsusb -d 0955:7c18

tao.wang · July 13, 2018, 12:03am

We tried on my colleague’s VM with the same cable and the same carrier board, and it works. Then I come back to my Ubuntu 16.04 Dell Laptop:

uninstall all Jetpacks
reinstall JetPack 3.2.1
put TX2 in recovery mode and lsusb to get the Nvidia
restart my laptop
do the flash, the same. The log file is:

###############################################################################

L4T BSP Information:

R28 (release), REVISION: 2.1, GCID: 11272647, BOARD: t186ref, EABI: aarch64,

DATE: Thu May 17 07:29:06 UTC 2018

###############################################################################
Error: probing the target board failed.
Make sure the target board is connected through
micro-B USB port and is in recovery mode.

linuxdev · July 13, 2018, 9:31pm

VMs rarely work unless you’ve set up USB and ethernet correctly (the default settings usually cause failure). On the other hand, if the VM settings work, then it validates that the Jetson is doing as expected. This in turn implies your laptop’s USB has some issues. About the only workaround is if you have a USB HUB which can be put between the laptop and Jetson…the PHY on the HUB might be able to get a signal when the PHY on the laptop does not.

tao.wang · July 13, 2018, 10:25pm

This really make sense. Anyway, I tested with a Belkin hub on two different carrier boards (including the developer kit), JetPack 3.2.1, Ubuntu 16.04, still the same.

linuxdev · July 13, 2018, 10:35pm

Unfortunately, that tends to imply your laptop is failing (probably hardware). On the other hand, you can test to see if the port itself is at least saying it is in USB2 mode. Run “lsusb -t” with something slow plugged in to that connector (e.g., a mouse or keyboard). The “root_hub” should show “480M” for USB2, or “5000M” for USB3. The mouse or keyboard would probably show “1.5M”. Then plugging in anything USB2 and cable of operating at slower speeds (e.g., a USB external hard drive) and see if the USB2 item shows “480M”. If not, then something capable of running slower would fall back to a slower interface, e.g., a USB2 hard drive showing as “1.5M” would be evidence the bus itself is having problem and dropping back to USB1.0 speeds (something like a joystick might show “12M” which is USB1.1).

Note that only some devices can function when dropping back to a slower speed. The Zed stereo camera cannot operate below USB3, so falling back to USB2 would be a failure condition. The Jetson in recovery mode is similar, but uses USB2…the Jetson cannot fall back to USB1.1, and perhaps the USB bus is in bad enough condition that it won’t function at 480Mbit/s speeds.

tao.wang · July 13, 2018, 11:00pm

Here is what I got when I plug in different things on the USB port on my laptop. It seems the hardware is ok. So there is something wrong in my Ubuntu 16.04. I use this laptop for almost all my work on it. It works with most of my USB devices.

No extra USB devices plugged in:
$ lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
|__ Port 3: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M
|__ Port 3: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M
|__ Port 4: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 5: Dev 7, If 0, Class=Video, Driver=uvcvideo, 480M
|__ Port 5: Dev 7, If 1, Class=Video, Driver=uvcvideo, 480M

Plug in dongle of wireless mouse
$ lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
|__ Port 1: Dev 14, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 1: Dev 14, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 3: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M
|__ Port 3: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M
|__ Port 4: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 5: Dev 7, If 0, Class=Video, Driver=uvcvideo, 480M
|__ Port 5: Dev 7, If 1, Class=Video, Driver=uvcvideo, 480M

Plug in a USB key
$ lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
|__ Port 1: Dev 14, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 1: Dev 14, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 2: Dev 15, If 0, Class=Mass Storage, Driver=usb-storage, 480M
|__ Port 3: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M
|__ Port 3: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M
|__ Port 4: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 5: Dev 7, If 0, Class=Video, Driver=uvcvideo, 480M
|__ Port 5: Dev 7, If 1, Class=Video, Driver=uvcvideo, 480M

linuxdev · July 14, 2018, 12:03am

This shows the port does get detected as USB2. Signal quality though can cause reverting to slower speeds, and I’m curious if the USB2 key is able to read and remain at USB2? Probably it can, but if you want to find out, then start this in one terminal:

watch -n 1 lsusb -t

Watch this in another terminal:

dmesg --follow

Then in another terminal, assuming the USB key shows up as “/dev/sdb” (adjust for your case) just cat the whole content to “/dev/null” and see if at any time during the operation USB re-enumerates or changes:

cat /dev/sdb > /dev/null

What it comes down to is that each USB device’s data is coming in over a balanced pair line, and must essentially have a clean enough “eye diagram” that the software does not detect errors (the eye diagram must be sufficiently “open” and jitter must not be too high). Every single device has different RF qualities, and so whether that eye diagram is an error or not most often won’t have anything to do with basic operation, nor will all failures be because of broken hardware. Simply changing the length of the cable can make an otherwise good or bad device become the opposite. I don’t suspect that the port hardware is broken, but I do suspect that this port does not have as good of an “eye diagram” as it should for this particular device. Obviously that port is good with most devices, but it seems you may be out of luck with the Jetson.

Software (and especially firmware) can change how well the port works even with the same signal quality. Ubuntu (or Debian) itself is unlikely to be a problem unless the firmware or specific driver for the PHY has a problem (the former is more likely than the latter).