An unstable USB connection causes the flashing to stop without exiting when AdbPush raises an error

Hello,

I am using L4T 38.4 Jetpack 7.1 for the Thor 5000 and I have noticed that the flashing script is stuck when any AdbPush-call is throwing an error. This could happen, when the usb connection is lost by a bad usb cable or faulty hardware.

['/media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/mfi_auvidea-p3834-0008-x242/unified_flash/out/bsp_images/tools/flashtools/bootburn/flash_bsp_images.py', '-b', 'jetson-t264', '--l4t', '-D', '-P', '/media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/mfi_auvidea-p3834-0008-x242/unified_flash/out/bsp_images/flash_workspace', '--l4t_boot_chain_select', 'A', '--usb-instance', '2-1.4']


Adb push failed -- /media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/mfi_auvidea-p3834-0008-x242/unified_flash/out/bsp_images/tools/flashtools/flash/adb -s 2U10U1118000007G6060 push die0_bctCopiesBlob.tmp  /tmp

Process Process-1:1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/mfi_auvidea-p3834-0008-x242/unified_flash/out/bsp_images/tools/flashtools/bootburn/../bootburn_t264_py/bootburn_adb.py", line 1257, in FlashUsingADB
    result = self.SendFileUsingADB(partitionInfo, SkipWriteZeroChunk, TempDir, queue, UpdatePartitions)
  File "/media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/mfi_auvidea-p3834-0008-x242/unified_flash/out/bsp_images/tools/flashtools/bootburn/../bootburn_t264_py/bootburn_adb.py", line 699, in SendFileUsingADB
    self.AdbPush(partitionInfo.FileName)
  File "/media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/mfi_auvidea-p3834-0008-x242/unified_flash/out/bsp_images/tools/flashtools/bootburn/../bootburn_t264_py/bootburn_adb.py", line 344, in AdbPush
    AbnormalTermination("Adb push failed -- " + adbCommand, nverror.NvError_Adb)
  File "/media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/mfi_auvidea-p3834-0008-x242/unified_flash/out/bsp_images/tools/flashtools/bootburn/../bootburn_t264_py/flashtools_nverror.py", line 260, in AbnormalTermination
    raise OSError(errorCode)
OSError: 53

I still need to evaluate py-spy output here. I have added it, because it might be a good starting point for your team to evaluate this issue.

root@auvidea-HP-Z620-Workstation:/media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra# py-spy dump --pid 724656
Process 724656: python3 /media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/unified_flash/out/bsp_images/tools/flashtools/bootburn/flash_bsp_images.py -b jetson-t264 --l4t -D -P /media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/unified_flash/out/bsp_images/flash_workspace --l4t_boot_chain_select A --usb-instance 2-1.4
Python v3.8.10 (/usr/bin/python3.8)

Thread 724656 (idle): "MainThread"
    poll (multiprocessing/popen_fork.py:27)
    wait (multiprocessing/popen_fork.py:47)
    join (multiprocessing/process.py:149)
    _exit_function (multiprocessing/util.py:357)
    _bootstrap (multiprocessing/process.py:318)
    _launch (multiprocessing/popen_fork.py:75)
    __init__ (multiprocessing/popen_fork.py:19)
    _Popen (multiprocessing/context.py:277)
    _Popen (multiprocessing/context.py:224)
    start (multiprocessing/process.py:121)
    ParallelFlashImages (bootburn_t264_py/bootburn_lib.py:2795)
    FlashImages (bootburn_t264_py/bootburn_lib.py:3236)
    flash_bsp_active (bootburn_t264_py/flash_bsp_images.py:119)
    run (multiprocessing/process.py:108)
    _bootstrap (multiprocessing/process.py:315)
    _launch (multiprocessing/popen_fork.py:75)
    __init__ (multiprocessing/popen_fork.py:19)
    _Popen (multiprocessing/context.py:277)
    _Popen (multiprocessing/context.py:224)
    start (multiprocessing/process.py:121)
    flash_bsp (bootburn_t264_py/flash_bsp_images.py:216)
    <module> (flash_bsp_images.py:40)
root@auvidea-HP-Z620-Workstation:/media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra# py-spy dump --pid 724066
Process 724066: python3 /media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/unified_flash/out/bsp_images/tools/flashtools/bootburn/flash_bsp_images.py -b jetson-t264 --l4t -D -P /media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/unified_flash/out/bsp_images/flash_workspace --l4t_boot_chain_select A --usb-instance 2-1.4
Python v3.8.10 (/usr/bin/python3.8)

Thread 724066 (idle): "MainThread"
    poll (multiprocessing/popen_fork.py:27)
    wait (multiprocessing/popen_fork.py:47)
    join (multiprocessing/process.py:149)
    _exit_function (multiprocessing/util.py:357)
    _bootstrap (multiprocessing/process.py:318)
    _launch (multiprocessing/popen_fork.py:75)
    __init__ (multiprocessing/popen_fork.py:19)
    _Popen (multiprocessing/context.py:277)
    _Popen (multiprocessing/context.py:224)
    start (multiprocessing/process.py:121)
    flash_bsp (bootburn_t264_py/flash_bsp_images.py:216)
    <module> (flash_bsp_images.py:40)
root@auvidea-HP-Z620-Workstation:/media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra# py-spy dump --pid 724058
Process 724058: python3 /media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/unified_flash/out/bsp_images/tools/flashtools/bootburn/flash_bsp_images.py -b jetson-t264 --l4t -D -P /media/auvidea/images/nvidia/nvidia_sdk/38.4_MASSFLASH_TEST/Linux_for_Tegra/unified_flash/out/bsp_images/flash_workspace --l4t_boot_chain_select A --usb-instance 2-1.4
Python v3.8.10 (/usr/bin/python3.8)

Thread 724058 (idle): "MainThread"
    poll (multiprocessing/popen_fork.py:27)
    wait (multiprocessing/popen_fork.py:47)
    join (multiprocessing/process.py:149)
    flash_bsp (bootburn_t264_py/flash_bsp_images.py:220)
    <module> (flash_bsp_images.py:40)

This issue might break a lot of automation scripts (including ours for our production line)

I this issue can be reliably be reproduced by removing the USB cable during one of the AdbPush-transfers

I am not quite sure where the exact issue is, but I was able to get the script to exit as expected when adding partitionWriterProcess.daemon = True in unified_flash/out/bsp_images/tools/flashtools/bootburn_t264_py/bootburn_adb.py

            # Start Partition writer process
            partitionWriterProcess = Process(target=self.StartPartitiionWriterProcess, args=(adbTaskQueue, ))
            partitionWriterProcess.daemon = True
            partitionWriterProcess.start()

I got to test this a lot faster by using the following flashing commands:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash <board_config> internal
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only <board_config> internal

Do you already got a chance to replicate this issue?

FYK, This also causes the massflash script to be stuck in an infinite loop, without exiting.

Sorry, want to clarify what is the exact key item to reproduce this issue?

Unplugging the USB cable during ADB-Push should be enough to replicate this issue

I don’t quite get it. How was the first time you hit this issue? You unplug the cable during flash?

The first time I have encountered this issue during a massflash instance. I am not sure what the cause of it was in this case.

I looked into the logs and tried to replicate this issue later on by unplugging the USB.

This is bad because if the unified flashing process doesn’t exit properly after an ADB push error in a massflash environment, the massflash job never finish

We built custom scripts around the massflash tarball to track which Jetsons have been flashed. This relies on a proper exit code

Ok, so you expect the massflash tool shall stop automatically instead of hang there when issue happened?

Yes. It already does that everywhere else, with AdbPush being the only exception I have encountered so far.

let us check what we could do here. Thanks for reporting.

Hi,

want to confirm these two points

  1. what environment of your host pc is?
  2. at which timing shall I unplug the USB cable?

never mind. able to reproduce the error now.

I just noticed that I am running 20.04, instead of a newer version.
Still great that you could reproduce this issue on your end too 🥳