Custom (minimal) rootfs for Jetson Nano wont boot

[jjsalzano]

Part 2

***************
*** 1790,2212 ****
  nvidia-l4t-weston                             
  nvidia-l4t-x11                                
  nvidia-l4t-xusb-firmware                      
- obconf                                        
- obexftp                                       
- obexpushd                                     
- obsession                                     
- oem-config                                    
- oem-config-gtk                                
- onboard                                       
- onboard-common                                
- openbox                                       
- openbox-lxde-session                          
- openbox-menu                                  
- openprinting-ppds                             
- openssh-client                                
- openssh-server                                
- openssh-sftp-server                           
- openssl                                       
- orca                                          
- os-prober                                     
- p11-kit                                       
- p11-kit-modules:arm64                         
- p7zip                                         
- p7zip-full                                    
- packagekit                                    
- packagekit-tools                              
- parted                                        
  passwd                                        
- patch                                         
- patchutils                                    
- pavucontrol                                   
- pciutils                                      
- pcmanfm                                       
- pcmciautils                                   
- perl                                          
  perl-base                                     
- perl-modules-5.26                             
- perl-openssl-defaults:arm64                   
- phantomjs                                     
- phonon4qt5-backend-vlc:arm64                  
- phonon4qt5:arm64                              
- pigz                                          
- pinentry-gnome3                               
- pkg-config                                    
- plymouth                                      
- plymouth-label                                
- plymouth-theme-ubuntu-logo                    
- policykit-1                                   
- policykit-1-gnome                             
- policykit-desktop-privileges                  
- poppler-data                                  
- poppler-utils                                 
- powermgmt-base                                
- ppp                                           
- pppconfig                                     
- pppoeconf                                     
- pptp-linux                                    
- printer-driver-brlaser                        
- printer-driver-c2esp                          
- printer-driver-min12xxw                       
- printer-driver-pnm2ppa                        
- printer-driver-ptouch                         
- printer-driver-sag-gdi                        
  procps                                        
- psmisc                                        
- publicsuffix                                  
- pulseaudio                                    
- pulseaudio-module-bluetooth                   
- pulseaudio-utils                              
  python                                        
- python-apt-common                             
- python-cairo:arm64                            
- python-dbus                                   
- python-gi                                     
- python-gobject-2                              
- python-gps                                    
- python-gtk2                                   
  python-jetson-gpio                            
- python-matplotlib-data                        
  python-minimal                                
- python-talloc                                 
- python-urwid                                  
  python2.7                                     
  python2.7-minimal                             
  python3                                       
- python3-apport                                
- python3-apt                                   
- python3-aptdaemon                             
- python3-aptdaemon.gtk3widgets                 
- python3-asn1crypto                            
- python3-blinker                               
- python3-brlapi                                
- python3-bs4                                   
- python3-cairo:arm64                           
- python3-certifi                               
- python3-cffi-backend                          
- python3-chardet                               
- python3-crypto                                
- python3-cryptography                          
- python3-cups                                  
- python3-cupshelpers                           
- python3-cycler                                
- python3-dateutil                              
  python3-dbus                                  
- python3-dbus.mainloop.pyqt5                   
- python3-debconf                               
- python3-debian                                
- python3-decorator                             
- python3-defer                                 
- python3-distro-info                           
- python3-distupgrade                           
- python3-distutils                             
- python3-feedparser                            
  python3-gi                                    
- python3-gi-cairo                              
- python3-html5lib                              
- python3-httplib2                              
- python3-icu                                   
- python3-idna                                  
  python3-jetson-gpio                           
- python3-jwt                                   
- python3-keyring                               
- python3-keyrings.alt                          
- python3-launchpadlib                          
- python3-lazr.restfulclient                    
- python3-lazr.uri                              
- python3-lib2to3                               
- python3-louis                                 
- python3-lxml:arm64                            
- python3-macaroonbakery                        
- python3-mako                                  
- python3-markupsafe                            
- python3-matplotlib                            
  python3-minimal                               
- python3-nacl                                  
- python3-numpy                                 
- python3-oauth                                 
- python3-oauthlib                              
- python3-pam                                   
- python3-pandas                                
- python3-pandas-lib                            
- python3-pkg-resources                         
- python3-problem-report                        
- python3-protobuf                              
- python3-pyatspi                               
- python3-pymacaroons                           
- python3-pyparsing                             
- python3-pyqt5                                 
- python3-pyqt5.qtsvg                           
- python3-pyqt5.qtwebkit                        
- python3-pyxattr                               
- python3-requests                              
- python3-requests-unixsocket                   
- python3-rfc3339                               
- python3-scipy                                 
- python3-secretstorage                         
- python3-simplejson                            
- python3-sip                                   
- python3-six                                   
- python3-software-properties                   
- python3-speechd                               
- python3-systemd                               
- python3-tk:arm64                              
- python3-tz                                    
- python3-uno                                   
- python3-update-manager                        
- python3-urllib3                               
- python3-urwid                                 
- python3-wadllib                               
- python3-webencodings                          
- python3-xdg                                   
- python3-xkit                                  
- python3-yaml                                  
- python3-zope.interface                        
  python3.6                                     
  python3.6-minimal                             
- qdbus                                         
- qml-module-org-kde-kquickcontrolsaddons:arm64 
- qml-module-qtmultimedia:arm64                 
- qml-module-qtquick2:arm64                     
- qt-at-spi:arm64                               
- qt5-gtk-platformtheme:arm64                   
- qtchooser                                     
- qtcore4-l10n                                  
- qttranslations5-l10n                          
- rdate                                         
- read-edid                                     
  readline-common                               
- remmina                                       
- remmina-common                                
- remmina-plugin-rdp:arm64                      
- remmina-plugin-secret:arm64                   
- remmina-plugin-vnc:arm64                      
- resolvconf                                    
- rfkill                                        
- rhythmbox                                     
- rhythmbox-data                                
- rhythmbox-plugin-alternative-toolbar          
- rhythmbox-plugin-zeitgeist                    
- rhythmbox-plugins                             
- rpcbind                                       
- rsync                                         
- rsyslog                                       
- rtkit                                         
- rtmpdump                                      
- runc                                          
- samba-libs:arm64                              
- sane-utils                                    
- scrot                                         
- seahorse                                      
  sed                                           
  sensible-utils                                
- session-migration                             
- session-shortcuts                             
- sgml-base                                     
  shared-mime-info                              
- shotwell                                      
- shotwell-common                               
- simple-scan                                   
- smplayer                                      
- smplayer-l10n                                 
- smplayer-themes                               
- snapd                                         
- software-properties-common                    
- software-properties-gtk                       
- sound-icons                                   
- sound-theme-freedesktop                       
- speech-dispatcher                             
- speech-dispatcher-audio-plugins:arm64         
- speech-dispatcher-espeak-ng                   
- spice-vdagent                                 
- squashfs-tools                                
- ssh-import-id                                 
- ssl-cert                                      
  sudo                                          
- system-config-printer                         
- system-config-printer-common                  
- system-config-printer-udev                    
- system-tools-backends                         
  systemd                                       
  systemd-sysv                                  
  sysvinit-utils                                
- t1utils                                       
- tango-icon-theme                              
  tar                                           
- tasksel                                       
- tasksel-data                                  
- thunderbird                                   
- thunderbird-gnome-support                     
- tk8.6-blt2.5                                  
- totem                                         
- totem-common                                  
- totem-plugins                                 
- transmission-common                           
- transmission-gtk                              
- ttf-bitstream-vera                            
- tzdata                                        
- ubiquity                                      
- ubiquity-casper                               
- ubiquity-frontend-debconf                     
- ubiquity-frontend-gtk                         
- ubiquity-frontend-kde                         
- ubiquity-ubuntu-artwork                       
- ubuntu-artwork                                
- ubuntu-desktop                                
- ubuntu-docs                                   
- ubuntu-drivers-common                         
- ubuntu-fan                                    
  ubuntu-keyring                                
  ubuntu-mono                                   
- ubuntu-release-upgrader-core                  
- ubuntu-release-upgrader-gtk                   
- ubuntu-report                                 
- ubuntu-session                                
- ubuntu-settings                               
- ubuntu-software                               
- ubuntu-sounds                                 
- ubuntu-system-service                         
- ubuntu-touch-sounds                           
- ubuntu-wallpapers                             
- ubuntu-wallpapers-bionic                      
- ubuntu-web-launchers                          
  ucf                                           
- udev                                          
- udisks2                                       
- unity                                         
- unity-asset-pool                              
- unity-control-center                          
- unity-greeter                                 
- unity-gtk-module-common                       
- unity-gtk2-module:arm64                       
- unity-gtk3-module:arm64                       
- unity-lens-applications                       
- unity-lens-files                              
- unity-lens-music                              
- unity-lens-photos                             
- unity-lens-video                              
- unity-schemas                                 
- unity-scope-calculator                        
- unity-scope-chromiumbookmarks                 
- unity-scope-colourlovers                      
- unity-scope-devhelp                           
- unity-scope-firefoxbookmarks                  
- unity-scope-home                              
- unity-scope-manpages                          
- unity-scope-openclipart                       
- unity-scope-texdoc                            
- unity-scope-tomboy                            
- unity-scope-video-remote                      
- unity-scope-virtualbox                        
- unity-scope-yelp                              
- unity-scope-zotero                            
- unity-scopes-master-default                   
- unity-scopes-runner                           
- unity-services                                
- unity-session                                 
- unity-settings-daemon                         
- uno-libs3                                     
- unzip                                         
- update-inetd                                  
- update-manager                                
- update-manager-core                           
- update-notifier                               
- update-notifier-common                        
- upower                                        
- ure                                           
- usb-modeswitch                                
- usb-modeswitch-data                           
- usbmuxd                                       
- usbutils                                      
- ussp-push                                     
- uthash-dev:arm64                              
  util-linux                                    
- uuid-dev:arm64                                
- va-driver-all:arm64                           
- vdpau-driver-all:arm64                        
- vim                                           
- vim-common                                    
- vim-runtime                                   
- vino                                          
- vlc-data                                      
- vlc-plugin-base:arm64                         
- vlc-plugin-video-output:arm64                 
- vulkan-utils                                  
- wamerican                                     
- wget                                          
- whiptail                                      
- whoopsie                                      
- whoopsie-preferences                          
- wireless-regdb                                
- wireless-tools                                
- wpasupplicant                                 
- x11-apps                                      
  x11-common                                    
- x11-session-utils                             
- x11-utils                                     
- x11-xkb-utils                                 
- x11-xserver-utils                             
- x11proto-core-dev                             
- x11proto-damage-dev                           
- x11proto-dev                                  
- x11proto-fixes-dev                            
- x11proto-xext-dev                             
- x11proto-xf86vidmode-dev                      
- xarchiver                                     
- xauth                                         
- xbitmaps                                      
- xbrlapi                                       
- xcursor-themes                                
  xdg-user-dirs                                 
- xdg-user-dirs-gtk                             
- xdg-utils                                     
- xdotool                                       
- xfonts-base                                   
- xfonts-encodings                              
- xfonts-scalable                               
- xfonts-utils                                  
- xinit                                         
- xinput                                        
  xkb-data                                      
- xloadimage                                    
- xmms2-core                                    
- xmms2-plugin-alsa                             
- xmms2-plugin-id3v2                            
- xmms2-plugin-mad                              
- xmms2-plugin-vorbis                           
- xorg                                          
- xorg-docs-core                                
- xorg-sgml-doctools                            
- xscreensaver                                  
- xscreensaver-data                             
- xserver-common                                
- xserver-xephyr                                
- xserver-xorg                                  
- xserver-xorg-core                             
- xserver-xorg-input-all                        
- xserver-xorg-input-libinput                   
- xserver-xorg-input-wacom                      
- xserver-xorg-legacy                           
- xserver-xorg-video-all                        
- xserver-xorg-video-amdgpu                     
- xserver-xorg-video-ati                        
- xserver-xorg-video-fbdev                      
- xserver-xorg-video-nouveau                    
- xserver-xorg-video-radeon                     
- xserver-xorg-video-vesa                       
- xterm                                         
- xtrans-dev                                    
- xul-ext-ubufox                                
- xwayland                                      
- xxd                                           
  xz-utils                                      
- yelp                                          
- yelp-xsl                                      
- youtube-dl                                    
- zeitgeist-core                                
- zeitgeist-datahub                             
- zenity                                        
- zenity-common                                 
- zip                                           
- zlib1g-dev:arm64                              
  zlib1g:arm64                                  
--- 275,310 ----

[linuxdev]

Once again, I’m just adding notes as I go through things. What follows is not specific.

This is long, the package I think is most relevant is “util-linux”. You can skip down to search for util-linux if you want to see the highest odds of something needed in the form of a package.

Also, you previously show what looks like “dmesg” output, but not a full serial console. “dmesg” starts with the kernel load, and this does verify the kernel is found and loaded, but I am wondering if you can provide a full serial console boot log?

Can you post the content of “extlinux.conf”?

An important question is to know how you installed the packages you have? Did you use QEMU on the host PC directly on the “Linux_for_Tegra/rootfs/” directorym or just via modifying packages for “apply_binaries.sh”? I’m interested to know if a tool like “apt” and “dpkg” were used such that the tools enforced dependencies when adding, removing, or modifying packages. It looks like you’ve added to the package list which “apply_binaries.sh” uses; if so, are there any errors when adding packages? Have you tried manually adding these packages via QEMU and apt? At one point early in Jetson history all additions were from a tar archive unpacking. At some point it started using QEMU and actual package tools, but I’m not sure how much currently uses QEMU since there is also a deb package “unpack” function which can use tar after unpacking. It would be good to post a full log of when “apply_binaries.sh” is run on an otherwise pristine “rootfs/” content (applied to whatever minimal rootfs you start with):
sudo ./apply_binaries.sh 2>&1 | tee binaries_log.txt

Also, consider that the system is booting such that the kernel loads and many things succeed, and what is failing is either the systemd services, prior to reaching a command line, or something else preventing command line. It seems initrd worked, modules are found, and the ext4 filesystem is available.

On the host PC, if you don’t have “tree” installed, then you might “sudo apt-get install tree”. Then, post the output of:

cd /etc/systemd
tree

---

Before I make suggestions on packages, consider creating a “current” package list for later diff:
dpkg -l | tee packages_start.txt

Then, log the actual install of any package (I’m assuming that you will do this in QEMU on the “Linux_for_Tegra/rootfs/” via edits to “apply_binaries.sh”, but if that is not how you manage packages, then let me know). Example, if you were to install package “binutils”:
sudo apt-get install binutils 2>&1 | tee install_binutils.txt
(“apply_binaries.sh” could be modified to use “tee” to log each package command)

Normally “tee” will delete a file before populating it unless you tell it to “append”. In the above example any previous content in “install_binutils.txt” would be deleted. You could simply have multiple commands append to that file, but it lacks any kind of empty line between installs:
sudo apt-get install binutils 2>&1 | tee -a install_binutils.txt
(notice the “-a” option for “append”)

I don’t know that binutils is actually needed, and probably it is not, but it is a good example.

I don’t see “util-linux” mentioned. This provides agetty and getty, and so this is high on the list of things which might break text mode login. This is the number 1 thing I’d consider adding as a test.

I doubt it is an issue, but sometimes gettext is used in init scripts.

“gvfs” might not be needed, especially since much of it is used by the GUI, but it is related to virtual filesystems, and might be involved in some user space failures. Not likely to cause a boot failure unless it is setting up a filesystem and stalls out due to this.

Here is one which catches my attention, but I don’t know if it really matters:
libdrm-tegra0
…which has this package description:
Userspace interface to tegra-specific kernel DRM services -- runtime
(note that there will be access to the GPU even if there is no GUI)

libefiboot1 might matter in the most recent releases using UEFI, but if it did, then I doubt the kernel would have ever loaded.

libelf1 I would consider a candidate because it can be used for reading ELF files. Not necessarily a problem, but it is conceivable that some init item uses this.

Not sure if libevent matters, but it might.

Mentioned earlier, maybe libfuse2 matters for user space.

You said this is headless, but I see some support for X and Wayland. Does anything change if, in your “extlinux.conf”, you put this at the end of the “APPEND” key/value pair?

plymouth is actually more involved in boot than many people realize. It is often called a splash screen package, but it is more than that. There is a chance that plymouth might be needed for reasons other than splash screen.

As long as there is USB there is a moderately strong chance that “udev” is needed.

Does anything change if you add this to the end of the extlinux.conf “APPEND” key/value pair?
systemd.unit=multi-user.target
(if you are going to text mode it should not matter, but if going to graphical mode, then it might)

Sometimes m4 is used in startup scripts.

[jjsalzano]

Thank you for all your help with this and the thorough notes.

So it turns out it was some packages that we were missing, however I am still not sure exactly which one yet, I still want to dig into it. I will be taking your notes and looking into it, Thank you very much.
I tried adding the util-linux package just by itself and this wasn’t quite enough, so there is still some investigations I have to do.

We were told about was this guide from some contacts we have:
https://nvidia-ai-iot.github.io/jetson-min-disk/minimal-l4t.html

This ended up working with the “nvubuntu-bionic-minimal-aarch64-packages” list. We are still hoping to cut another couple hundred MB off of the final image size but we are at least down to 2.2GB from the initial 5.6GB.

[linuxdev]

If you do complete this and can post a package list (I don’t know if your list would be proprietary or not) or any tips on how you learned to cut down on packages it would be appreciated to post what you can. Other people probably would appreciate any kind of insight the experience produces.