As a follow-up I did manage to get the system to POST and have it boot up.
As for what is happening exactly, I’m at a complete and total loss but I’m pretty sure its hardware related.
I think the memory card reader either came with something wedged between its connectors and the card which wasn’t noticeable, or it is intermittently defective in which case the issue will recur.
I’m about 25 hours into this project now and 90% sure this is a defective memory reader RMA. Unfortunately I still am without a response from anyone on the Jetson Support Team. I’ve documented my efforts below in the hopes they’ll respond and move forward with the replacement RMA. Hopefully it will helps someone out along the line way as well.
What I find troubling is, of all the below tests I’ve run under linux, they all failed on the Jetson Nano up until I moved setups/location and attempted to boot with the Windows 7 created media. It was probably coincidence and by that point I’ve inserted and removed the Micro SD card at least 20-30 times. After that first successful boot the exact same tests with the previous images work (boots on the jetson, but with nominal errors/warnings).
For reference I’ve started this odyssey running Ubuntu 18.04.4 LTS to create the initial media.
Nvidia provides a link to the sdcard image which is ~ 5.2GiB in size, name and sha256sum listed below. Nvidia doesn’t provide a hash to confirm integrity after download but that’s not where this problem was (though it is a best practice to weed out transmission errors).
The image size uncompressed is 12GiB, 12 884 901 888.
The image is 12GiB and media sizing at this point for existing products only come in multiples of 8, 16, 32, 64GiB for Micro SD.
Writing the image with balenaEtcher on Ubuntu 18.04.4 LTS to an ONN 16 or 32GiB micro sd card succeeds, but results in a confused kernel after it attempts to validate without first removing and reinserting the card after the write, there also appears to be some small amount of file corruption depending on the reader being used when this occurs (DigiPower Walmart/SuperTop) and no file corruption with other readers(Merkury/GenesysInc 05e3:0751).
Checking the APP partition with fsck shows a long list of underlying inode problems from the original image. Allowing fsck to correct the errors doesn’t appear to have any effect on the outcome. (i.e. It still doesn’t POST at this time)
Writing the image with dd instead of Etcher on Ubuntu 18.04.4 LTS to the same media succeeds, but results in the same errors/warnings from kernel about GPT but no confused kernel and same behavior when testing above with exception to the write errors which don’t happen when you use a non-SuperTop USB card reader and after you remove / reinsert the removable media which alters a partition table (standard practice).
Replacing the media with a new Micro SD card has no impact. Previous tests fail.
Correcting the GPT headers for different sized media removes the warnings/errors the kernel throws but the device still appears dead when booting from the jetson nano. Correcting GPT headers and running fsck to fix the superblock have no change in behavior on the jetson nano.
Writing the image with balenaEtcher on Windows 7 to the same media succeeds, and the system will boot up normally without any issue (at this point).
Comparing the fresh-preboot image created on Windows 7 vs Linux shows significant deviation with binary differences in the two image files when comparing raw bytes using ‘cmp’ and inspected at those addresses with ‘hd’. The input files used pre-Etcher between linux and windows were verified identical.
The first difference starts with several GPT header subfields for CRC/backup LBA locations, and a few others, and numerous/too many to count differences after 14680537, LBA/sector 28672 (* 512 byte LBA), i.e. the APP partition superblock.
Retrying all of the above tests that had failed (with the Merkury USB reader) all succeed with nominal errors after the first successful boot with Windows 7 created media, and the jetson boots.
Testing without a memory card in the device shows the exact same symptoms as I was experiencing, and the memory card was previously seated correctly in the Jetson.