Driver crashes after upgrade from 510.68.02 to 515.48.07 on GTX 1080ti

It’s on Arch Linux with stock kernel 5.18.3. It doesn’t matter if driver is loaded after system boot or during boot.
On 510.68.02 everything worked fine.

nvidia-bug-report.sh script never finishes, I attached partial output.
nvidia-bug-report.log.gz (766.7 KB)

Relevant dmesg output:

[ 7755.567880] nvidia 0000:14:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 7755.698996] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  515.48.07  Fri May 27 03:18:00 UTC 2022
[ 7755.700096] [drm] [nvidia-drm] [GPU ID 0x00001400] Loading driver
[ 7756.351783] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:14:00.0 on minor 1
[ 7756.362363] nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint.
[ 7756.370912] nvidia-uvm: Loaded the UVM driver, major device number 503.
[ 7768.396285] nvidia-modeset: ERROR: GPU:0: Display engine push buffer channel allocation failed: 0x65 (Call timed out [NV_ERR_TIMEOUT])
[ 7768.410230] nvidia-modeset: ERROR: GPU:0: Failed to allocate display engine core DMA push buffer
[ 7784.451875] BUG: kernel NULL pointer dereference, address: 0000000000000070
[ 7784.451880] #PF: supervisor read access in kernel mode
[ 7784.451881] #PF: error_code(0x0000) - not-present page
[ 7784.451882] PGD 0 P4D 0 
[ 7784.451884] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 7784.451886] CPU: 8 PID: 123052 Comm: Xorg Tainted: P        W  OE     5.18.3-arch1-1 #1 2090c6f1d9d20f39bd14c0acb6fa89ddb994d43f
[ 7784.451889] Hardware name: ASUS System Product Name/ROG CROSSHAIR VIII HERO (WI-FI), BIOS 4201 04/26/2022
[ 7784.451890] RIP: 0010:_nv002521kms+0x18/0x70 [nvidia_modeset]
[ 7784.451913] Code: ff c6 44 24 2f 01 eb af 66 2e 0f 1f 84 00 00 00 00 00 41 54 55 49 89 fc 53 89 d5 41 b8 04 00 00 00 ba 02 01 02 00 48 83 ec 10 <8b> 46 70 8b 3d cf cd 0f 00 48 8d 4c 24 0c 89 ee 89 44 24 0c e8 bf
[ 7784.451915] RSP: 0018:ffffb78162cefc30 EFLAGS: 00010286
[ 7784.451916] RAX: 0000000000000000 RBX: 0000000020020000 RCX: 0000000000006c08
[ 7784.451918] RDX: 0000000000020102 RSI: 0000000000000000 RDI: ffff9f69a0fc7008
[ 7784.451919] RBP: 0000000000010009 R08: 0000000000000004 R09: 00000000fffffffe
[ 7784.451919] R10: 0000000000000000 R11: 0000000000000001 R12: ffff9f69a0fc7008
[ 7784.451921] R13: ffff9f69a0fc70a0 R14: 0000000000000fff R15: 0000000000010008
[ 7784.451922] FS:  00007fafe5f0e100(0000) GS:ffff9f6f2ec00000(0000) knlGS:0000000000000000
[ 7784.451923] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7784.451924] CR2: 0000000000000070 CR3: 0000000a66862000 CR4: 0000000000350ee0
[ 7784.451926] Call Trace:
[ 7784.451927]  <TASK>
[ 7784.451929]  _nv002520kms+0xb3/0x150 [nvidia_modeset 658855af3f998e4c93b01ca095c666f654e238c7]
[ 7784.451948]  _nv002294kms+0x4da/0x720 [nvidia_modeset 658855af3f998e4c93b01ca095c666f654e238c7]
[ 7784.451967]  ? __check_object_size+0x143/0x160
[ 7784.451971]  ? _nv000448kms+0xa0/0xa0 [nvidia_modeset 658855af3f998e4c93b01ca095c666f654e238c7]
[ 7784.451987]  _nv000633kms+0x34/0x50 [nvidia_modeset 658855af3f998e4c93b01ca095c666f654e238c7]
[ 7784.452003]  nvKmsIoctl+0x96/0x1d0 [nvidia_modeset 658855af3f998e4c93b01ca095c666f654e238c7]
[ 7784.452019]  nvkms_ioctl+0x118/0x180 [nvidia_modeset 658855af3f998e4c93b01ca095c666f654e238c7]
[ 7784.452036]  nvidia_frontend_unlocked_ioctl+0x3c/0x50 [nvidia cdb2ec842bb5797c80be4ec5b9ce9c833bd49f74]
[ 7784.452259]  __x64_sys_ioctl+0x91/0xc0
[ 7784.452262]  do_syscall_64+0x5f/0x90
[ 7784.452265]  ? __x64_sys_ioctl+0x91/0xc0
[ 7784.452267]  ? syscall_exit_to_user_mode+0x26/0x50
[ 7784.452268]  ? do_syscall_64+0x6b/0x90
[ 7784.452270]  ? nvidia_frontend_unlocked_ioctl+0x3c/0x50 [nvidia cdb2ec842bb5797c80be4ec5b9ce9c833bd49f74]
[ 7784.452479]  ? __x64_sys_ioctl+0x91/0xc0
[ 7784.452481]  ? syscall_exit_to_user_mode+0x26/0x50
[ 7784.452482]  ? do_syscall_64+0x6b/0x90
[ 7784.452484]  ? do_syscall_64+0x6b/0x90
[ 7784.452485]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 7784.452487] RIP: 0033:0x7fafe68727af
[ 7784.452489] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 18 48 8b 44 24 18 64 48 2b 04 25 28 00 00
[ 7784.452490] RSP: 002b:00007fffacc80fb0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 7784.452492] RAX: ffffffffffffffda RBX: 00000000c0106d00 RCX: 00007fafe68727af
[ 7784.452493] RDX: 00007fffacc81010 RSI: 00000000c0106d00 RDI: 0000000000000012
[ 7784.452494] RBP: 00007fffacc81010 R08: 00007fffacc805a0 R09: 00007fffacc805bc
[ 7784.452495] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000012
[ 7784.452495] R13: 00007fffacc81060 R14: 0000558a47236530 R15: 00007fafe528e880
[ 7784.452497]  </TASK>
[ 7784.452498] Modules linked in: nvidia_uvm(POE) nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) cdc_acm rfcomm snd_seq_dummy snd_hrtimer snd_seq overlay wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel bridge stp llc nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 cmac algif_hash algif_skcipher nf_tables af_alg nct6775 hwmon_vid nfnetlink bnep lm92 xfs sch_cake iwlmvm snd_hda_codec_realtek intel_rapl_msr vboxnetflt(OE) intel_rapl_common mac80211 snd_hda_codec_generic vboxnetadp(OE) libarc4 ledtrig_audio snd_hda_codec_hdmi snd_hda_intel edac_mce_amd btusb vboxdrv(OE) snd_intel_dspcfg snd_usb_audio btrtl snd_intel_sdw_acpi pkcs8_key_parser btbcm snd_usbmidi_lib iwlwifi snd_hda_codec btintel pktcdvd asus_ec_sensors i2c_dev uvcvideo kvm_amd btmtk snd_rawmidi eeepc_wmi iwlmei snd_hda_core videobuf2_vmalloc videobuf2_memops snd_seq_device bluetooth snd_hwdep videobuf2_v4l2 asus_wmi snd_pcm cfg80211
[ 7784.452531]  videobuf2_common kvm sparse_keymap r8169 snd_timer videodev ecdh_generic realtek platform_profile igb sp5100_tco vfat snd joydev mdio_devres rfkill mousedev fat rapl mc video pcspkr wmi_bmof mxm_wmi k10temp i2c_piix4 soundcore crc16 libphy mei dca mac_hid pinctrl_amd acpi_cpufreq dm_multipath kvmfr(OE) ipmi_devintf ipmi_msghandler sg crypto_user fuse lzo_rle zram ip_tables x_tables btrfs blake2b_generic xor raid6_pq libcrc32c crc32c_generic dm_crypt cbc encrypted_keys trusted asn1_encoder tee hid_logitech_hidpp hid_logitech_dj dm_mod amdgpu drm_ttm_helper uas usb_storage ttm usbhid crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel tpm_crb crypto_simd gpu_sched nvme sr_mod cryptd tpm_tis ccp drm_dp_helper xhci_pci tpm_tis_core cdrom nvme_core xhci_pci_renesas tpm wmi rng_core vfio_pci vfio_pci_core irqbypass vfio_virqfd vfio_iommu_type1 vfio tcp_bbr
[ 7784.452571] CR2: 0000000000000070
[ 7784.452573] ---[ end trace 0000000000000000 ]---
[ 7784.452573] RIP: 0010:_nv002521kms+0x18/0x70 [nvidia_modeset]
[ 7784.452592] Code: ff c6 44 24 2f 01 eb af 66 2e 0f 1f 84 00 00 00 00 00 41 54 55 49 89 fc 53 89 d5 41 b8 04 00 00 00 ba 02 01 02 00 48 83 ec 10 <8b> 46 70 8b 3d cf cd 0f 00 48 8d 4c 24 0c 89 ee 89 44 24 0c e8 bf
[ 7784.452593] RSP: 0018:ffffb78162cefc30 EFLAGS: 00010286
[ 7784.452594] RAX: 0000000000000000 RBX: 0000000020020000 RCX: 0000000000006c08
[ 7784.452595] RDX: 0000000000020102 RSI: 0000000000000000 RDI: ffff9f69a0fc7008
[ 7784.452596] RBP: 0000000000010009 R08: 0000000000000004 R09: 00000000fffffffe
[ 7784.452596] R10: 0000000000000000 R11: 0000000000000001 R12: ffff9f69a0fc7008
[ 7784.452597] R13: ffff9f69a0fc70a0 R14: 0000000000000fff R15: 0000000000010008
[ 7784.452598] FS:  00007fafe5f0e100(0000) GS:ffff9f6f2ec00000(0000) knlGS:0000000000000000
[ 7784.452599] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7784.452600] CR2: 0000000000000070 CR3: 0000000a66862000 CR4: 0000000000350ee0

Maybe you encountered this?

Don’t mind the repo name, happens with both prop and open source kernel driver on 5.18 kernel.

I have AMD CPU, not Intel, so this is not applicable for me.

I am seeing the same crash with 5.18.3 kernel
Trying to get A5500 working under linux which requires 515 as minimum.