Deploying with L4T/flash.sh breaks on my carrier board

Hi Undertow10,

These should help debug this issue.
bootloader/partner/common/drivers/usbf/class/transport/tegrabl_transport_usbf.c
bootloader/partner/common/drivers/usbf/xusbf/tegrabl_xusbf.c -> tegrabl_usbf_enumerate()

Please note that device tree setting for usb configuration like uphy mapping should not affect the flash process. If it can affect the flash process, then we won’t have those cases like “my device is able to boot up but my usb port does not work”.

If you still want to debug from software side, you could compare the uart log from your carrier board (OK) and the NG carrier board.

Hi Wayne,

Thanks for the info. I found the files you mentioned in the older CBoot source download, but the common files must be shared with the CPU Recovery Bootloader? I was able to locate some of the error print-outs, but not all. Can you share the complete source code for this project? Or at least, so you know how I can turn on a higher level of error logging (to print pr_debug, etc)?

Hi Undertow10,

Sorry that I cannot reveal the source code.

How about sharing

  1. your hardware difference between your own carrier board and devkit?

  2. The patch you made and I can help build cpu recovery bootloader binary for you.

Can you build a debug version of the CPU recovery bootloader binary with all logging turned on? Or is there a way for me to enable it? It’s difficult to know where the error is originating from without seeing more messages. Presuming the common usb code is the same as the CBoot source I have, the error I’m getting is:

[0013.605] I>  SUPER SPEED
[0013.628]  > XUSBF: Failed to configure enumeration.
[0013.633] E> usbf_enum failed err = 0x2f2f0c02

Which I think, if printed using TEGRABL_ERROR_PRINT(), would print:

C> ERROR: Highest Layer Module = 0x5e, Lowest Layer Module = 0xbc, Aux Info = 0xc, Reason = 0x2
// Reason 0x2 = TEGRABL_ERR_INVALID

But that error gets produced in a lot of places, so it’s unclear where it originates.

Here are the hardware changes on the TX2 connector from one board to the other. We switched from TX1 to a TX2-4GB in this revision, so we implemented the M.2 slot for Wifi, mostly based off the DevKit schematics.

These are the first boards we’ve had professionally assembled. I will try building one board by hand with only the necessary components for flashing and see if that works.


I have narrowed down the problem by removing hardware. The issue seems to be that we now have an internal USB hub chip connected to PEX_RFU. The old (working) board had the USB hub SS lines connected to USB_SS0, but we had to move them to PEX_RFU because we’re not allowed to use PEX1 and USB_SS0 at the same time. Only the SS connections moved, but the USB2 lines remain on USB1_D+/USB1_D-.

I can remove the USB hub chip from the PCB and the flash process works correctly.

What is it about having our USB Hub’s SS lines attached to PEX_RFU that cause TBoot to not enumerate USB properly? Why did it work properly when the USB Hub was attached to USB_SS0 lines? Is TBoot checking for a certain condition on the PEX_RFU lines? Can this functionality be disabled in a custom build of TBoot?

This poster also seems to have a similar unresolved issue: https://devtalk.nvidia.com/default/topic/1028438/can-not-flash-sh-anything-when-the-usb-hub-which-connect-to-pex_rfu-usb3_1-working-/

Hi Wayne,

Do you have any updates on this? It looks like the USB Hub chip is holding the RX line high and sending a low pulse every ~12 ms. Why would this cause a problem with TBoot on the PEX_RFU lines, but not the USB_SS0 lines?

Hi,

Sorry for late reply. I am checking this issue with other engineers internally.

Undertow10,

I think we need to know more about #26.

According to #26, below is my understanding, could you confirm?

  1. The working board:
    USB Hub + USB_SS0+ USB1 -> working
    No Hub + USB_SS0+ USB1 -> working

  2. The NG board:
    USB hub + PEX_RFU +USB1 -> NG
    No Hub + PEX_RFU+ USB1 -> working

Yes, that is correct.

I can also confirm that if the SS RX lines (SS TX out from hub) are held low (USB hub held in reset) the flash process works normally.

The signal appears to be some sort of polling / detection sequence from the USB hub. It seems TBoot is seeing a signal on those lines and getting confused. The USB Hub IC is hard-wired onto the same PCB, so we have no way to disconnect the SS lines.

Thank you Undertow10!!! You definitely win the prize for checking out the hubs. Fortunately, we designed our carrier with a way to control the resets to the hubs we have on our carrier. Today we rewrote the code so that if the board is powered up with the force recover switch held then the hubs are held in reset until the board is power cycled. As you can guess from your experience the Flash routine now works.

Here are some of my conclusions for anyone else with similar problems.

We noticed that we can flash original NV config2 materials on our board. Our config5 changes failed to flash correctly. In some ways it appeared to be related to the config5 changes to the USB3 ports.
We tried a number of reconfigurations of the

.cfg files and the dts files without success (although I wonder if there may be a possible solution in that direction! but we are out of time for working on that approach.) (Question is there a best way to associate USB2-1 and USB3-1 as belonging together and does anyone think that might have helped?)

In any case it appears (seeing Undertow10’s experience) that any config that disables USB3-0 and enables USB3-1 (at a minimum) sets up a situation where the flash may not work. If you have changed from config2 and use a hub on USB3-1 (and perhaps USB3-2 or USB3-0) you may experience problems flashing. Design in a way to hold the pins mentioned by Undertow10 low. Holding the hub in reset is a great way to accomplish that.

In any case, we have a solution on our board. Good luck if you are working on the same issue. I’ll be watching to see if some better solutions come to light.

Thanks again to WayneWWW, Undertow10, and all others that have been helping to solve this.

Best regards,
crveit

Hi Crveit,

Thanks for your post. To clarify, with our board it does not matter which ODMDATA configuration we are using, the flash operation always fails on the PCB with the hub SS lines attached to PEX_RFU. As Wayne has indicated, the device tree is not in use at this point, so it should have no effect. It is my understanding that the ODMDATA configuration value only affects the device tree through plugin-manager, but I am not 100% sure. If, when you say “configuration” you mean your hardware changed to that layout, then yes, any configuration with a hub on PEX_RFU would fail.

I’m curious to hear what code you wrote to hold the hub in reset when the force recover button is pushed. Is this code running on the Jetson? Or do you have some sort of intermediary processor between the buttons and your Jetson?

Hi Undertwo10,

I think changing ODMDATA may be worth trying in the beginning. Actually, ODMDATA not only affects device tree but also has some magic in BPMP and causes uphy lane mapping change. (device tree indeed will not have any effect since kernel is not yet up)

However, I am not sure crevit’s comment that disable USB3-1 would make it work. You could try it first.

Hi Undertow10, We have a Cortex m4 processor that does a few minor monitoring tasks on our carrier and fortunately it was already monitoring the force-recovery switch and was wired to reset the hubs. The code I was refering to was for that processor. I am not sure that it would be possible to use the TX2i to manage those resets during a force-recovery. If so it would probably require an intimate knowledge of the entire boot sequence at a deeper level than I seem to see in the various guides.

Did I correctly understand that you can not flash the original config2 code? That would indicate some differences in the symptoms on our carriers. Hope you can find a fix without respinning the board. But if you do need to do so there are a number of inexpensive processors that could hold the resets low. You could also capture the force-recovery status with a flip-flop if worst came to worst (in our current fix we hold the hub resets low until a power cycle). Although I hope you can find a simpler way.

Moving to WayneWWW’s comment: Wayne, given we can flash the config2 materials, is it possible to give the config2 ODMDATA initially ( for flashing ) and then somehow change it? (Either after the fact with rewriting that partition, or perhaps even changing it in a later part of the boot?) Such a method might help us give a work around for board that are already in the field.

Best regards,
Crveit

Hi crevit,

Sorry that I didn’t follow all your thread previously since there are multiple users here.

We need t clarify each person’s case. For Undertow10, we already knew that he puts a HUB on PEX_RFU and USB1.

Could you also briefly introduce your current status?

Hi Undertow10,

Can you pls share the schematics of usb hub chip on USB_SS1 lines and on USB_SS0 lines?

Hi Trumany,

The USB Hub chip on our non-functional boards is on PEX_RFU pins, not USB_SS1 or USB_SS0. We do have another device on USB_SS1. USB_SS0 is unused, as we need those lanes for PEX1 instead.

Are you asking for our schematic of the working board or the non-working board?

What specifically are you looking to learn? I posted the TX2 pinout portion of our schematics in post #25.

sorry typo, should be config USB_SS#1 = PEX_RFU port. We are investigating why the usb hub on PEX_RFU lines will cause flashing on USB0 port fail as you said. If you can share the connection of this hub board, will be helpful to understand.

Our schematic is attached. I’ve edited a few pages together to simplify the viewing.