Let's make flashing from virtual machines reliable

[rgovostes]

The advice is often given here to not flash a device from a virtual machine. I can share anecdotally that I have succeeded a few times from an Ubuntu VM on an Intel-based Mac, but I’ve never gotten it to work on an Apple Silicon Mac (fails immediately), both using VMware Fusion. There are some other stories linked from this thread.

Obviously, a lot of people want to use virtual machines. They’re incredibly convenient, and one doesn’t need to keep a special laptop around with the blessed version of Ubuntu just in case there’s a need to reflash. So rather than discouraging it, it would be beneficial to try to identify and resolve some of the root causes of problems that arise.

A reason that’s been given is that,

VMs are not supported is because they don’t handle USB correctly for the case of a USB device which repeatedly disconnects and reconnects (which is what happens during flash)

I am surprised about this because it cannot be universally true across all virtualization programs, host operating systems, and hypervisors.

It would be good to know more specifics about which programs have issues and what part of the process breaks down. For example, is there some state that gets messed up on the Jetson device when it gets passed from the host to the VM? Is there some additional latency that breaks some time-sensitive operation?

[linuxdev]

The issue is that every VM has different methods of configuration. It is up to the person installing the VM to make sure USB is passed through correctly, and there is no way that the flash software can know about being in a VM. Even if the flash software knows it is in a VM, the flash software does not have access to reconfiguring the VM (consider that often a VM is used for security to purposely prevent software inside of it from modifying what is outside). Any question on setup generally involves asking the VM author rather than the flash software. Flash software simply expects this to be an “ordinary” system where USB and other services work (losing USB and not regaining it is a case of not working even if it is configurable as a fix rather than as a hardware failure or bug).

[cyato]

I tested with both Virtualbox (6.x and 7.0) and VMware Workstation Pro on Windows 10 host. I did not test with VMware Fusion on Mac M1/M2.

The results;

• Virtualbox (6.x and 7.0) on Windows 10 → Flash fails.
• VMware Workstation Pro 16 on Windows 10 → Flash succeeds.
• VMware Fusion → [Have not tested]

Here is the issue that I saw with Virtualbox.

1. tegraflash.py after all the other checking, make Jetson reboot in recovery mode.
2. Jetson reboots, and the host (Windows10) detects the USB disconnect
3. However, maybe because the Jetson comes back too quickly, the VM (Ubuntu) does NOT detect the USB disconnect, keep listing the old USB device instance
   • Possibly, this is the Virtualbox mechanism to combat unstable USB connection on some other device (some people reports loop of disconnect and reconnect)
4. When the VM (Ubuntu) tries to communicate with the USB device, it does not have the latest USB device so fails

I did not see this issue with VMware Workstation Pro 16.
VMware Workstation Pro 16 finds the re-connected USB device properly and completed the whole flashing flow without an issue.

So one workaround for VirtualBox is that the script on the VM waits for enough time after rebooting JAX into recovery mode, instructing the user to manually plug out the Type-C USB cable for flashing and put the cable back, and make sure the device is again seen on the VM (ex. through lsusb).
With this workaround/hack, I got to flash a JAX with the flash.sh command in the Linux_for_Tegra directory set up by the SDK Manager.

The modification I added to the tegraflash_internal.py is like this.

def tegraflash_reboot(args):
    if args[0] == 'coldboot':
        info_print('Coldbooting the device')
    elif args[0] == 'recovery':
        info_print('Rebooting to recovery mode **** [ Within the next 30sec, manually plug out the USB cable for flashing and put the cable back in ] ****')
    else:
        raise tegraflash_exception(args[0] + " is not supported")

    if int(values['--chip'], 0) == 0x21 or int(values['--chip'], 0) == 0x18:
        try:
            command = exec_file('tegradevflash')
            command.extend(['--reboot', args[0]])
            run_command(command)
            time.sleep(2)
        except:
            tegraflash_tboot_reset(args)
    else:
        if check_ismb2():
            tegraflash_tboot_reset(args)
            info_print('Rebooting to recovery mode =========================')
            time.sleep(30)
        else:
            command = exec_file('tegradevflash')
            command.extend(['--reboot', args[0]])
            run_command(command)

The info messages may not always come on the SDK Manager terminal, so you basically need to just react to the Windows device disconnect sound.

[rgovostes]

Thanks @cyato! Very useful information.

Today I performed a successful flash from an ARM64 Ubuntu virtual machine on VMware Fusion Tech Preview 22H2 on an Apple Silicon Mac, possibly the first ever from an ARM64 host?

*** The target t186ref has been flashed successfully. ***
Reset the board to boot from internal eMMC.

Several tools that come with the L4T BSP are compiled for the i386 architecture and to run them on ARM64 they need to be executed under emulation, using the qemu-i386 user mode emulator. However, by default qemu-i386 loads these binaries at a base offset that causes them to behave irregularly and sometimes crash. You must modify the configuration to use -B 0. (Also, I used the latest QEMU, rather than a much older version that Ubuntu packages.)

I had no issues with the USB disconnect and reconnecting, and I didn’t otherwise have to make any modifications to the BSP scripts.