Unable to setup Headless Operation on Jetson Nano TX2 on Custom Carrier Board

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
,
1

Hi there
I have a eMMC Jetson Nano (ie, not SD card model) installed in my custom carrier board. A colleague has the custom carrier board responding properly, so I suspect my issues are not related to the custom carrier board.
I have used SDK Manager to flash the system (Jetpack 4.4.1 - I am advised that Jetpack 4.5 is not suitable for the application, so I haven’t tried that yet) over the USB micro link. SDK Manager fails at the point where the SDK files are to be installed, since at that point (I believe) the SDK Manager requires “the first boot” setup which is customarily done with KVM directly on the Jetson Nano. The custom carrier board has no HDMI pins connected since the application is headless.

For the “first boot” I can see in Linux and Windows10 (lsusb and Device Manager) the L4T device.

  • Linux: The ‘screen’ serial command is unresponsive. Attempts to ssh are “refused.”
  • Windows: I can see COM4 instantiated with the correct VID/PID. I can see a CDC NCM USB device which has a bang for ‘no driver.’ When I try to ‘update driver’ and point Windows to the .inf on the L4T drive, Windows says there are no acceptable drivers.
    In Windows, I PuTTY to COM4, but the console is unresponsive, even if I tap the space bar.
    In Windows, if I PuTTY to ssh on nvidia | ubuntu@192.168.55.1 the ssh connection is “refused.”
    In Windows, I can ping 192.168.55.1. Regarding the Jetson Ethernet port, I can ping it no problem from Windows or Linux (got the IP address from my LAN router) - it also refuses ssh connections, which I imagine is expected since I have not yet completed the “first boot” over the micro-USB setup.

I am attaching the boot log from the UART. I note:

  • failed I2C
  • detects a HDMI device.
    These two things seem anomalous to me. If Jetson Nano detects the HDMI device does it shutdown the serial UART setup? Is that what’s happening? Any ideas why the I2C might be “failing”? The custom board has some I2C devices, but no external pullups, so should be OK. uartbootlog.txt (43.9 KB)

On the forums, they seem to suggest I have the wrong Jetpack image. Please link me to the correct image I should be using.

Thanks in advance.

3

Maybe you can configure the user account first and then check your issue again. The behavior may change after your configure your user account.

4

Not sure how that is done given the problem statement mentions no serial link achieved, Is there some other way?

5

What I cannot understand is why do you say your board detects a HDMI device and also there is no HDMI pins?

6

If your board does not have HDMI port, do not directly flash your board with sdkmanager.

There is no 100% guarantee that sdkmanager can work on custom board. The device tree and kernel from sdkmanager are for nv devkit only.

For custom board, every developers here should modify our default device tree to match their board.

7

Please specify on what basis the Jetson Nano decides there is a HDMI device present.

8

Thanks Wayne. In the documentation, it seems fairly clear that as of Jetpack 4.4, there are two pathways supported.

  1. Headless operation, and
  2. Normal keyboard/video/mouse operation.
    So I am not sure why the Jetpack would be unsuitable since it (ostensibly) natively supports headless development.
9

Also please note. Our custom carrier board is design very similarly to the Jetson Carrier board, essentially a stripped-down version.

10

Hi,

The “headless” is not the problem here. The logic of headless on devkit is just “this board can work fine even with no display”. The display driver is still in the background to detect monitor hotplug event.

However, on your board, this function is causing unexpected display event.
If you don’t have any display, please disable all nvdisplay in your device tree and try again.

Our custom carrier board is design very similarly to the Jetson Carrier board, essentially a stripped-down version.

We always hear such comment from every users here. But issues are still a lot, so there is no guarantee.

11

Thank you. Where do I find my “device tree?”

12

Please check the page to download kernel page. After rebuilding kernel, dtb files will be genearted too.
https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3231/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fkernel_custom.html%23wwpID0E0SD0HA

Please note, modifying device tree is very common for custom board.

13

Thanks again. Are you confirming that my Jetson Nano is not providing the micro-USB serial session for “first boot” because it is detecting a HDMI device?
In that case, please help me troubleshoot this by specifying what qualifies the presence of a HDMI device for the Jetson Nano.

14

My comment is still same. Disable the display driver from device tree first as my previous comment.

15

Thanks again. Recall in the initial statement, a colleague has the system completing “first boot” through the serial port. This is on the same custom carrier board.
To assist us in identifying the source of the disparate behaviour (his works properly, mine doesn’t) it would be useful to know under which mechanism the Jetson Nano is choosing not to open the serial port “first boot” session.
To my knowledge, once in “first boot” we can configure things such as whether the graphics library should be enabled. We would of course disable the graphics library and open only ssh (over Ethernet and micro-USB). That’s the plan. That’s working on the other board. That’s what the tutorials seems to be saying.
Evidence in the boot log seems to indicate that the Jetson Nano “thinks” there is a HDMI device attached, and on that basis does not open the serial session for “first boot.” Could I please be directed to documentation on the topic:

  • what steers whether the serial session if opened for “first boot”
  • what specific hardware signals qualify that a HDMI device is present.

Thanks for your previous advice about rebuilding the device tree. I believe this is the true Production option. For now, for the initial application hardware bringup phase, we need the shorter route to work. Your assistance with the specific questions would be appreciated.
Thanks again.

16

OK, so I found the HDMI_HPD signal on the nVidia carrier board, on pin 96. On my custom board, I ties this high (1.8V) with a 10k resistor. Unfortunaltey, this hasn’t changed the boot log - still reports “HDMI detected” Weird.

Anyways, I looked again at the boot log, (attached in previous post) - check out the lines #.

#399 → [ 0.000000] bootconsole [uart8250] enabled
#421 → [ 16.760579] Please complete system configuration setup on desktop to proceed…

This indicates that the system is indeed waiting for a serial setup session.
Are we back to the Windows CDC NCM in Device Manager, with the bang - indicating missing driver error?
Do we have a fix for this? The .inf on the L4T drive reports as ‘not a compatible driver.’
Perhaps if I can get the driver installed I will be able to proceed with the serial session “first boot.”
What baud rate should I be expecting if other than 115200?

Thanks again.

So it seems that I should be getting a session on the UART for 'first boot."

17

  1. Please go to Linux_for_Tegra on your host machine, go to rootfs/boot/extlinux/extlinux.conf, remove the “quiet” keyword inside it. Then, reflash the board and it will give you more log from serial console.

  2. Actually, we don’t use Windows to debug the serial console. If you already has sdkmanager, it means you have a ubuntu host there. Please directly use that ubuntu host to check the serial console.

  3. I don’t deal with hardware issue. But my colleague @Trumany can. Please share what you’ve done to the HDMI_HPD.

18

Hi again. This link strongly suggests that I should be able to run the “first boot” headless:
https://developer.nvidia.com/embedded/learn/get-started-jetson-nano-2gb-devkit#setup
So this does not seem like a “device tree” problem when flashing the Jetson Nano development environment from SDKManager.

To get around this, I did use my custom board to flash the Jetson Nano development environment from SDKManager. Of course this fails at the “install SDK” step since it’s prior to that when a “first boot” is required. At least the Ubuntu OS is flashed. I then removed the Jetson Nano module from my custom board into an nVidia P3450 carrier board.
In the nVidia carrier board, the “first boot” by serial also fails through SDKManager. So I reluctantly plugged in a monitor/keyboard/mouse, booted, and went through the “first boot” in the Ubuntu GUI - no worries. I then removed my Jetson Nano module from the nVidia carrier, back to my custom carrier board.
Now, on my custom carrier board I can ssh using either the micro-USB, or the Ethernet port (both are working), after having completed the “first boot” on an nVidia carrier board in the GUI.

Also, just addressing a previous query in the thread, I have tried using “screen” in Linux for the initial serial “first boot” - Linux sees the port in lsusb, and can see the L4T, but the serial session does not start and is not responsive. Similar to the Windows experience described previously.

That is a real pain that the Jetson Nano module is not accepting the initial serial connection. I would like some visibility into why this might be the case. Am I flashing an incorrect OS to my module? In SDK Manager I choose “Jeston Nano.” My module is https://www.arrow.com/en/products/900-13448-0020-000/nvidia
I am referencing this guide: Jetson Nano - Headless Setup - JetsonHacks

Is there something wrong with the Jetpack 4.5 that prevents “first boot” by serial session over micro-USB?

19

Hi,

I think something need to be clarified here. Let me reply your comment one by one. It seems you put many issues altogether.

This link strongly suggests that I should be able to run the “first boot” headless:
So this does not seem like a “device tree” problem when flashing the Jetson Nano development environment from SDKManager.

Yes, we support to configure user account headless. But you said your custom board “detects the hdmi even when there is no physical HDMI”. Actually, I am not sure if this headless configuration would come up or not if you connect a monitor on it. I mean the system/driver thinks you connect one.
Thus, I asked you to disable monitor detection first. The easiest and fastest way for me to disable HDMI detection on any board if modifying the device tree. Since everyone has some categories that they are more familiar with, I think it is ok to modify the hardware part in hpd pin. However, because it didn’t work on your side, I suggest to disable the HDMI by device tree again.

----> This part is for the “hdmi detection when no physical hdmi connected”.

To get around this, I did use my custom board to flash the Jetson Nano development environment from SDKManager. Of course this fails at the “install SDK” step since it’s prior to that when a “first boot” is required. At least the Ubuntu OS is flashed. I then removed the Jetson Nano module from my custom board into an nVidia P3450 carrier board.

This one sounds no problem to me.

Also, just addressing a previous query in the thread, I have tried using “screen” in Linux for the initial serial “first boot” - Linux sees the port in lsusb, and can see the L4T, but the serial session does not start and is not responsive. Similar to the Windows experience described previously.
That is a real pain that the Jetson Nano module is not accepting the initial serial connection. I would like some visibility into why this might be the case. Am I flashing an incorrect OS to my module? In SDK Manager I choose “Jeston Nano.” My module is https://www.arrow.com/en/products/900-13448-0020-000/nvidia
I am referencing this guide: Jetson Nano - Headless Setup - JetsonHacks

Is there something wrong with the Jetpack 4.5 that prevents “first boot” by serial session over micro-USB?

Because you described this problem like a universal one that every one would hit, I have to tell you that:
Actually, we can configure it through serial console on our host machines well. Thus, it is more like some issues happened between nano and your host side.

Maybe you can share the log from host dmesg when you try to open the oem-config. So that we can tell what is going on. Also, share the detail commands you are using would be better.

21

Hi again
I think I have the reason. First, some more information.
I noticed that some nVidia Jetson Nano cards booted fine on our carrier board, offering the initial boot (and subsequent boot) session over microUSB. Other nVidia Jetson Nano cards on our same carrier board failed to boot properly. The performance of the nVidia Jetson Nano modules as to whether is would boot properly or not, seemed to be consistent, although I am pretty sure there was at least one occasion where the behaviour changed. To my recollection, all of our Jetson Nano modules booted properly on an nVidia carrier board P3450.
We discovered that our UART2 (pins 236, 238) RX pin was not pulled up to 1.8V on our carrier board. There should be no need. However, we installed a pullup of 47k, and immediately those nVidia Jetson Nano boards that consistently failed to boot properly on our carrier board, commenced booting properly.
Could I please get an explanation why there is this dependency on the RX pin of UART2? Is this dependency documented?
My conclusion is that the floating UART2 RX pin was sometimes read as ‘IDLE’ and sometimes as ‘BREAK,’ depending on the way the wind blows. This indeterminate RX state caused the variation in booting experience.
Perhaps there is a Jetson Jetpack code fix that could be implemented to remove this dependency. Either, remove the dependency, or implement a pullup resistor function on that device input, if that’s an option for that pin.

23

What’s your nano module version? Is it A02 or B01? If A02, there should be no pull-up or pull-down on the pin as it is one of strapping pins of ram code selection. If B01, no matter to add pu or pd as there is buffer to isolate the pin status during power on.