Nvidia driver hangs the CPU for some time when booting. Motherboard: G1 Sniper Z87, Processor: i7-....

This problem occurs with regular Ubuntu 16.04 too. The Ubuntu logo hangs for some time, then I hear a beep sound from the tower, then the boot process continues. This happens every time I boot.

The problem doesn’t occur with the open-source drivers (the xorg drivers).

here’s part of dmesg.

[   40.036012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [nvidia-smi:1034]
[   40.036013] Modules linked in: nvidia_uvm(POE) intel_rapl x86_pkg_temp_thermal nvidia_drm(POE) intel_powerclamp input_leds nvidia_modeset(POE) coretemp nvidia(POE) kvm_intel snd_usb_audio snd_usbmidi_lib kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel mxm_wmi aes_x86_64 snd_hda_codec_hdmi crypto_simd snd_hda_codec_ca0132 snd_ca0106 snd_hda_intel snd_ac97_codec snd_hda_codec snd_hda_core snd_hwdep snd_seq_midi snd_seq_midi_event glue_helper snd_rawmidi cryptd ipmi_devintf ac97_bus snd_pcm snd_seq snd_seq_device snd_timer intel_cstate intel_rapl_perf ipmi_msghandler snd shpchp lpc_ich mei_me mei mac_hid soundcore serio_raw wmi parport_pc ppdev lp parport binder_linux(OE) ashmem_linux(OE) autofs4 hid_generic usbhid hid i915 i2c_algo_bit psmouse drm_kms_helper syscopyarea
[   40.036069]  sysfillrect sysimgblt alx ahci fb_sys_fops mdio libahci drm video
[   40.036086] CPU: 1 PID: 1034 Comm: nvidia-smi Tainted: P           OE    4.15.0-30-generic #32~16.04.1-Ubuntu
[   40.036086] Hardware name: Gigabyte Technology Co., Ltd. G1.Sniper Z87/G1.Sniper Z87, BIOS F4 11/21/2014
[   40.036254] RIP: 0010:os_io_read_dword+0xc/0x10 [nvidia]
[   40.036254] RSP: 0018:ffffaac702397bd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff11

Attached is the full dmesg output and the output of the debug script.
nvidia-bug-report.log.gz (148 KB)
dmesg.txt (92.2 KB)

bump!?

Looks like the hang is triggered by nvidia-smi, so the question is what starts that and why. Maybe start by checking your systemd units and udev rules.

Like I said, this bug happens on an Ubuntu 16.04 and on KDE neon. In both cases the Nvidia driver is installed through the repo (this https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa). Just the regular repo install.

I’m not sure what to look for. Isn’t the nvidia-smi running just because I have the proprietary drivers installed? Here are some systemd unit files:

/lib/systemd/system$ ll | grep nvidia
-rw-r--r--  1 root root   241 Mar 28 23:37 nvidia-persistenced.service
-rw-r--r--  1 root root   197 Jan 16  2015 nvidia-prime.service
/lib/systemd/system$ cat nvidia-persistenced.service 
[Unit]
Description=NVIDIA Persistence Daemon
Wants=syslog.target

[Service]
Type=forking
ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --no-persistence-mode --verbose
ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced
:/lib/systemd/system$ cat nvidia-prime.service 
[Unit]
Description=Enable NVIDIA GPU for PRIME on Shutdown
DefaultDependencies=no

[Service]
Type=oneshot
ExecStart=-/bin/sh -c 'echo ON > /proc/acpi/bbswitch'

[Install]
WantedBy=shutdown.target

as for udev rules, I checked the directory “/usr/lib/udev/” but it doesn’t exist, and the directory “/etc/udev/rules.d/” but it doesn’t seem to have anything that’s Nvidia related

I’ve taken a look at the driver package, nvidia-smi is started by the udev rule in
/lib/udev/rules.d/71-nvidia.rules
to create the /dev nodes.
IIRC, ubuntu doesn’t ship the suid nvidia-modprobe which would be used otherwise. Maybe check if commenting out the line udev rules file will still create the nodes.
From my experience, using nvidia-smi for that purpose will in rare cases lead to that hang you’re having, but don’t ask me why.

Thank you so much for the answer. This solved it. I just commented the line

ACTION=="add" DEVPATH=="/module/nvidia" SUBSYSTEM=="module" RUN+="/usr/bin/nvidia-smi"

and the delay went away.

Sorry for the very late reply.