Find command causes crash on Jetson AGX xavier

[ldonayre]

Hello,

Running the ‘find’ command with ‘-exec’ or a pipe to ‘xargs’ on my AGX Xavier causes it to crash and reboot immediately:

find / -type f  | xargs grep graphical.target
find / -type f  | xargs grep foo
The search word doesn't matter, any searched word will do.

The find command by itself causes no crash

find / -type f

A find of all files listed to a script and a subsequent ‘grep graphical.target’ prefixed to each entry does not cause a crash.

So the crash is specific to the “xarg” and the “exec” used in conjunction with the find command.

This ticket noted an issue related to ethernet related crash which was due to defective HW.
AGX Xavier easy to crash when ethernet network connected

I tried my test (find) with box off network (no ethernet) and also after systemctl set-default multi-user.target (runlevel 3) and the find crash persists.

My xavier is on my desktop, air conditioned room 68 degrees F. no vibrations, never dropped.

I’ve attached the output of the nvidia-bug-report-tegra.sh scripts.

nvidia-bug-report-tegra.log (429.6 KB)

[linuxdev]

@ldonayre

find / -type f | xargs grep graphical.target

What if you use the arguments intended for odd file names:
find / -type f -print0 | xargs -0 grep graphical.target

I suspect that some name with a space or other quoting requirement is getting in the way. This should not cause a crash, and a serial console log from just prior to the command, and up until reboot (serial console log would survive since it is logged from the host PC), would be useful. Knowing if the “-print0” to find and “-0” to xargs helps would immediately tell if the problem is dependent upon file names.

Further, you should probably also use the “-xdev” option in find to prevent reading “/sys”, “/proc”, and “/dev”. The option says to not cross filesystems (other mounts which are mounted on another mount). Knowing if that helps (without the “-print0” and “-0” options) would imply the reboot is from reading a non-filesystem file (e.g., reading a device special file or a kernel driver file). Example:
find / -xdev -type f -print0 | xargs -0 grep graphical.target
…or to purposely allow errors in odd file names to see if it is specifically a non-rootfs file issue:
find / -xdev -type f | xargs grep graphical.target

[ldonayre]

Thank you so much @linuxdev for taking the time on a Saturday to help me. Your suggestions are excellent and I appreciate them and used them to test futher and to find a successful workaround below.

Findings

# crashes
find / -type f -print0 | xargs -0 grep graphical.target

# works, does NOT crash
find / -xdev -type f -print0 | xargs -0 grep graphical.target

# works, does NOT crash
find / -xdev -type f | xargs grep graphical.target

Helpful! I have installed with and without patching (apt upgrade) and find fails unless -xdev is used.

In the case of the crashing test, no messages were on the console except the sudden termination of messages (which makes me think some part of this is affecting the kernel). The last entries are :

...
grep: /sys/kernel/slab/:tA-0000192/cgroup/cred_jar(76:systemd-modules-load.service)/alloc_calls: Function not implemented
grep: /sys/kernel/slab/:tA-0000192/cgroup/cred_jar(76:systemd-modules-load.service)/free_calls: Function not implemented
grep: /sys/kernel/slab/kmem_cache_node/alloc_calls: Function not implemented
grep: /sys/kernel/slab/kmem_cache_node/free_calls: Function not implemented
grep: /sys/kernel/debug/17000000.gv11b/fifo/profile/enable: Permission denied
grep: /sys/kernel/debug/tegra_hsp/ring: Permission denied

Addition fruitful findings:

I dual boot this Xavier between eMMC and an NFS mount (ttp and my own initrd) and noted that ‘find /’ worked correctly on my NFS variant.
My NFS rootfs is non-graphical (init 3) with many of the desktop/GUI pakcages removed (detail listing below) and the ‘find / …’ works without crashing with or without the -xdev.

So I did the following:

I rebuilt a new pristine rootfs from baseline image Tegra186_Linux_R32.5.1_aarch64.tbz2 (built samplefs, apply_binaries.sh, etc…), and used the resulting rootfs to flash the eMMC and I copied that same rootfs to the NFS export.

Both the eMMC and NFS mounted roots continued to crash via the ‘find / …’ command unless “-xdev” is specified.

I then upgraded both (eMMC and NFS roots) via “apt upgrade” and both of these rootfs continued to fail with the find command unless “-xdev” specified.

So not a network issue, but a package one it seems to me.

Solution - albeit by removing the ubuntu desktop

This works for me and the ‘find /’ (without “-xdev”) no longer crashes, but it removes the desktop.

sudo systemctl set-default multi-user.target

init 3

apt purge -y adwaita-icon-theme aisleriot alsa-utils aspell-en at-spi2-core bamfdaemon baobab bluez bluez bluez-obexd brltty brltty cheese-common chromium-browser chromium-browser chromium-codecs-ffmpeg-extra clipit colord compiz-gnome compiz-gnome crda deja-dup desktop-file-utils desktop-file-utils dictionaries-common dictionaries-common dosfstools eject eog espeak-ng-data:arm64 evince evince evince-common evolution-data-server evolution-data-server-common example-content example-content fakeroot ffmpeg file-roller fontconfig fontconfig-config fontconfig-config gdm3 gnome-disk-utility gnome-initial-setup gnome-keyring gnome-power-manager gnome-screensaver gnome-settings-daemon gnome-software gnome-system-tools gpg-agent gpsd-clients humanity-icon-theme ibus imagemagick indicator-applet indicator-application indicator-appmenu indicator-bluetooth indicator-datetime indicator-keyboard indicator-sound kio kwin-common:arm64 libcompizconfig0:arm64 libglade2-0:arm64 libgtk2.0-0:arm64 libgtk-3-0:arm64 libkf5crash5:arm64 libkf5globalaccel5:arm64 libqt5gui5:arm64 libreoffice-core libsane1:arm64 libsm6:arm64 libunity-core-6.0-9:arm64 libunity-misc4 libxaw7:arm64 libxft2:arm64 libxss1:arm64 lightdm-gtk-greeter light-themes lxlock lxpanel lxpolkit lxsession mpv nautilus network-manager network-manager-gnome network-manager-gnome network-manager-pptp notification-daemon notify-osd obsession oem-config oem-config-debconf oem-config-gtk onboard openbox openbox-lxde-session orca pcmanfm phonon4qt5:arm64 plymouth-theme-ubuntu-logo policykit-1-gnome pppconfig pppoeconf printer-driver-pnm2ppa pulseaudio rhythmbox rhythmbox-data samba-libs:arm64 sane-utils seahorse shotwell-common speech-dispatcher system-config-printer system-config-printer-common thunderbird totem transmission-gtk ubuntu-desktop ubuntu-drivers-common ubuntu-mono ubuntu-sounds unity unity-asset-pool unity-control-center unity-greeter unity-settings-daemon update-manager update-notifier vlc-plugin-base:arm64 vlc-plugin-video-output:arm64 whiptail whoopsie whoopsie-preferences whoopsie-preferences wireless-regdb wireless-tools wireless-tools wpasupplicant wpasupplicant x11-apps x11-common x11proto-dev x11-session-utils x11-xserver-utils xbrlapi xdg-user-dirs-gtk xfonts-base xfonts-scalable xfonts-utils xinit xscreensaver xscreensaver-data xserver-common xserver-xorg xserver-xorg-core xserver-xorg-legacy xterm yelp yelp-xsl youtube-dl zeitgeist-datahub

apt autoremove

init 6

Now ‘find / …’ works and does not cause a crash, but again no desktop which solves my issue, but may be a factor for others. I’m going to test against other Xaviers next week and see if I can reproduce this on other Desktop installs.

[linuxdev]

Since the “-xdev” option prevented crash it implies the crash was a side effect of trying to read “/sys” pseudo files not intended for read (I suspect if you manually “cat” whichever “/sys” file was hitting this error, then it would crash every time without ever using “find”, and this would be considered normal and expected behavior).

In cases where “-xdev” is not used (but still no crash) it might be that the specific pseudo files no longer exist, or else have some other configuration making an attempt to read non-fatal. Typically, if using find against “/”, I would always use “-xdev” to avoid pseudo files. Imagine for example you were to grep “/dev/random” for a word, and the random number file never ends? Or similar with “/dev/zero”…it would be a very long day of grep! :P

Note: An additional possible problem of crossing into pseudo files, imagine if you read from a watchdog timer file. You could do something unexpected with enabling or disabling this (files in “/dev” are also pseudo files and not real files).