TX2 USB Recovery Mode flash hangs

I’m using tegraflash.py to flash partitions on a custom TX2 device but I’m experiencing hangs most of the time when I attempt this operation.

I’m developing on Ubuntu 16.04 on a Dell XPS 15 9570.

I modified tegraflash_internal.py to prompt me to unplug and replug my USB cabled when it detects an error response of ‘BootRom is not running’ which helps to overcome this issue.

The output of my session is as follows:

Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0038 ] Generating RCM messages
[   0.0045 ] tegrarcm_v2 --listrcm rcm_list.xml --chip 0x18 --download rcm mb1_recovery_prod.bin 0 0
[   0.0052 ] RCM 0 is saved as rcm_0.rcm
[   0.0054 ] RCM 1 is saved as rcm_1.rcm
[   0.0054 ] List of rcm files are saved in rcm_list.xml
[   0.0054 ] 
[   0.0054 ] Signing RCM messages
[   0.0060 ] tegrasign_v2 --key None --list rcm_list.xml --pubkeyhash pub_key.key
[   0.0067 ] Assuming zero filled SBK key
[   0.0096 ] 
[   0.0096 ] Copying signature to RCM mesages
[   0.0103 ] tegrarcm_v2 --chip 0x18 --updatesig rcm_list_signed.xml
[   0.0113 ] 
[   0.0113 ] Boot Rom communication
[   0.0120 ] tegrarcm_v2 --chip 0x18 --rcm rcm_list_signed.xml
[   0.0126 ] BootRom is not running
[   1.3111 ] 
[   1.3111 ] Unplug and replug USB cable then press ENTER...

[  86.0185 ] BootRom is not running
[  87.2951 ] 
[  87.2951 ] Unplug and replug USB cable then press ENTER...

[  97.3799 ] BR_CID: 0x81801001644036470800000011fd8440
[  97.5315 ] RCM version 0X180001
[  97.5480 ] Boot Rom communication completed
[  98.5550 ] 
[  98.5577 ] tegrarcm_v2 --isapplet
[  98.5602 ] Applet version 01.00.0000
[  98.5780 ] 
[  98.5808 ] tegrasign_v2 --key None --getmode mode.txt
[  98.5833 ] Assuming zero filled SBK key
... flash succeeds ...

In the above log you can see the prompt that I added requesting an unplug and replug - at this point I physically unplug the USB cable to my device and then plug it back in and the tegrarcm_v2 command will succeed after one or more replug attempts.

I’ve also tested against the standard Jetson TX2 Developer Kit with the JetPack 3.3 flashing utilities with similar results. With enough replug attempts I can get the flashing to work.

I’ve also captured USB traffic via usbmon and Wireshark. When flashing fails I see a URB_BULK in request which returns a URB with length 0 and an -ENOENT error code. When it succeeds the URB length is 16.

Example capture for failure:

Request (raw):

0000   80 aa 27 42 ee 88 ff ff 53 03 81 27 01 00 2d 3c   .ª'Bî.ÿÿS..'..-<
0010   fd 97 75 5c 00 00 00 00 a4 8d 07 00 8d ff ff ff   ý.u\....¤....ÿÿÿ
0020   60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   `...............
0030   00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00   ................

Request (text):

Frame 174: 64 bytes on wire (512 bits), 64 bytes captured (512 bits) on interface 0
USB URB
    [Source: host]
    [Destination: 1.39.1]
    URB id: 0xffff88ee4227aa80
    URB type: URB_SUBMIT ('S')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x81, Direction: IN
    Device: 39
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: not present ('<')
    URB sec: 1551210493
    URB usec: 495012
    URB status: Operation now in progress (-EINPROGRESS) (-115)
    URB length [bytes]: 96
    Data length [bytes]: 0
    [Response in: 175]
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000200
    Number of ISO descriptors: 0

Response (raw):

0000   80 aa 27 42 ee 88 ff ff 43 03 81 27 01 00 2d 00   .ª'Bî.ÿÿC..'..-.
0010   fe 97 75 5c 00 00 00 00 b4 08 08 00 fe ff ff ff   þ.u\....´...þÿÿÿ
0020   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0030   00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00   ................

Response (text):

Frame 175: 64 bytes on wire (512 bits), 64 bytes captured (512 bits) on interface 0
USB URB
    [Source: 1.39.1]
    [Destination: host]
    URB id: 0xffff88ee4227aa80
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x81, Direction: IN
    Device: 39
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1551210494
    URB usec: 526516
    URB status: No such file or directory (-ENOENT) (-2)
    URB length [bytes]: 0
    Data length [bytes]: 0
    [Request in: 174]
    [Time from request: 1.031504000 seconds]
    [bInterfaceClass: Unknown (0xffff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000200
    Number of ISO descriptors: 0

Example capture for success:

Request (raw):

0000   c0 a9 27 42 ee 88 ff ff 53 03 81 29 01 00 2d 3c   À©'Bî.ÿÿS..)..-<
0010   5e 98 75 5c 00 00 00 00 9b 63 0b 00 8d ff ff ff   ^.u\.....c...ÿÿÿ
0020   60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   `...............
0030   00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00   ................

Request (text):

Frame 464: 64 bytes on wire (512 bits), 64 bytes captured (512 bits) on interface 0
USB URB
    [Source: host]
    [Destination: 1.41.1]
    URB id: 0xffff88ee4227a9c0
    URB type: URB_SUBMIT ('S')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x81, Direction: IN
    Device: 41
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: not present ('<')
    URB sec: 1551210590
    URB usec: 746395
    URB status: Operation now in progress (-EINPROGRESS) (-115)
    URB length [bytes]: 96
    Data length [bytes]: 0
    [Response in: 465]
    [bInterfaceClass: Vendor Specific (0xff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000200
    Number of ISO descriptors: 0

Response (raw):

0000   c0 a9 27 42 ee 88 ff ff 43 03 81 29 01 00 2d 00   À©'Bî.ÿÿC..)..-.
0010   5e 98 75 5c 00 00 00 00 da 63 0b 00 00 00 00 00   ^.u\....Úc......
0020   10 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00   ................
0030   00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00   ................
0040   40 84 fd 11 00 00 00 08 47 36 40 64 01 10 80 81   @.ý.....G6@d....

Response (text):

Frame 465: 80 bytes on wire (640 bits), 80 bytes captured (640 bits) on interface 0
USB URB
    [Source: 1.41.1]
    [Destination: host]
    URB id: 0xffff88ee4227a9c0
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_BULK (0x03)
    Endpoint: 0x81, Direction: IN
    Device: 41
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1551210590
    URB usec: 746458
    URB status: Success (0)
    URB length [bytes]: 16
    Data length [bytes]: 16
    [Request in: 464]
    [Time from request: 0.000063000 seconds]
    [bInterfaceClass: Vendor Specific (0xff)]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000200
    Number of ISO descriptors: 0
Leftover Capture Data: 4084fd11000000084736406401108081

I there a way that I can ensure a more reliable flash? Anyone else run into this issue?

Do you follow our OEM design guide? I didn’t see other users need to do such change in tegraflash before…

I believe we do - I’m new to this product and don’t know the exact details of the design.

Other developers on my team are able to connect reliably to perform a recovery mode flash with similar host systems.

Other developers on my team are able to connect reliably to perform a recovery mode flash with similar host systems.

Do you mean only your device have such problem?

Yes, two other developers are able to flash reliably with their host environments which are running Ubuntu 16.04 and one of them is even using the same make/model laptop as I am - Dell XPS 15 9570.

As a test, are the other developers able to flash your board? Or can your laptop flash one of their boards?

I ran a simple testing using a Ubuntu 16.04 live CD on a Panasonic FZ-G1 Toughpad and was able to get the flashing to work.

Tomorrow I will try the Live CD on my laptop and see if it’s hardware issue or an issue with my configuration.

A word of warning: A live CD/DVD distribution does not support loopback, and the base file system type is Fuse, not ext4. This will not be able to create a valid flash. You could install packages via JetPack this way, but flash itself is not possible under those circumstances (it might flash, but what you flash won’t work).

An external disk or USB drive formatted ext4 would work for space, but the live DVD kernel would still fail to be able to run certain loopback operations.

Thanks for the warning. It appears that the Ubuntu 16.04 live image supports loopback and is using it to to mount a squashfs root file system and then use aufs to provide a union between the base rootfs (read-only) and a writable upper file system.

It appears to work - but I’ll inspect the results more closely to make sure the images are being flashed correctly.

That being said, when I tested my live image on my laptop that was not working I was able to flash reliably. This leads me to believe that there is a difference in configuration between the live image and the installed image on my laptop.

So far as I know squashfs will not correctly support permissions as needed. I suspect you will get an apparent successful flash, and then a boot which doesn’t work correctly. If you use an external disk which is ext4 formatted as the location where you run the flash (it’d have to be fairly large), then it might work.