After playing with nouveau for a while I decided to go back to the binary driver - mainly due to vulkan support. Unfortunately the issue is still there with Fedora 31, 435.21 driver and 5.3.7 kernel:
[ 16.454126] ------------[ cut here ]------------
[ 16.454127] kernel BUG at /tmp/akmodsbuild.XCdjp5Qz/BUILD/nvidia-kmod-435.21/_kmod_build_5.3.7-301.fc31.x86_64/nvidia-drm/nvidia-drm-connector.c:439!
[ 16.454144] invalid opcode: 0000 [#1] SMP PTI
[ 16.454157] CPU: 6 PID: 736 Comm: systemd-udevd Tainted: P OE 5.3.7-301.fc31.x86_64 #1
[ 16.454170] Hardware name: CLEVO P150HMx/P150HMx, BIOS 4.6.4 08/09/2011
[ 16.454188] RIP: 0010:nv_drm_get_connector+0xa2/0x1d0 [nvidia_drm]
[ 16.454199] Code: 64 64 d0 ff 48 8b 44 24 20 65 48 33 04 25 28 00 00 00 0f 85 30 01 00 00 48 83 c4 28 4c 89 e8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 <0f> 0b 48 8d 7c 24 10 49 c7 c5 f4 ff ff ff e8 2b 64 d0 ff 49 8b 46
[ 16.454259] RSP: 0018:ffffb58a4056fa78 EFLAGS: 00010297
[ 16.454306] RAX: ffff8ac1f7c80068 RBX: 0000000000000008 RCX: 0000000000000002
[ 16.454354] RDX: ffff8ac1f7c80080 RSI: 0000000000000297 RDI: 0000000000000297
[ 16.454402] RBP: 0000000000000000 R08: ffff8ac21a38ca78 R09: 0000000000000000
[ 16.454451] R10: 000000000000000f R11: ffff8ac1f7f37c80 R12: ffff8ac1f7f38a88
[ 16.454497] R13: ffff8ac1f7c80068 R14: ffff8ac21a38c800 R15: 0000000000000001
[ 16.454544] FS: 00007f6d66bf1940(0000) GS:ffff8ac223380000(0000) knlGS:0000000000000000
[ 16.454626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 16.454677] CR2: 00007f4b20a872d0 CR3: 000000031d5d4005 CR4: 00000000000606e0
[ 16.454726] Call Trace:
[ 16.454776] nv_drm_add_encoder+0x19d/0x300 [nvidia_drm]
[ 16.454823] nv_drm_load+0x264/0x307 [nvidia_drm]
[ 16.454869] ? nv_drm_master_drop+0x60/0x60 [nvidia_drm]
[ 16.454927] drm_dev_register+0x111/0x150 [drm]
[ 16.454977] nv_drm_probe_devices+0x196/0x220 [nvidia_drm]
[ 16.455024] ? 0xffffffffc04e2000
[ 16.455073] do_one_initcall+0x46/0x1f4
[ 16.455119] ? _cond_resched+0x15/0x30
[ 16.455164] ? kmem_cache_alloc_trace+0x162/0x220
[ 16.455210] ? do_init_module+0x23/0x230
[ 16.455254] do_init_module+0x5c/0x230
[ 16.455302] load_module+0x27b1/0x2990
[ 16.455350] ? __do_sys_finit_module+0xaa/0x110
[ 16.455396] __do_sys_finit_module+0xaa/0x110
[ 16.455448] do_syscall_64+0x5f/0x1a0
[ 16.455497] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 16.455544] RIP: 0033:0x7f6d67c4115d
[ 16.455590] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d fb 5c 0c 00 f7 d8 64 89 01 48
[ 16.455718] RSP: 002b:00007fff90ac23a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 16.455805] RAX: ffffffffffffffda RBX: 0000565418c08bc0 RCX: 00007f6d67c4115d
[ 16.455853] RDX: 0000000000000000 RSI: 0000565418c12b90 RDI: 0000000000000017
[ 16.455902] RBP: 0000000000020000 R08: 0000000000000000 R09: 0000000000000007
[ 16.455949] R10: 0000000000000017 R11: 0000000000000246 R12: 0000565418c12b90
[ 16.455997] R13: 0000000000000000 R14: 000056541893f290 R15: 0000565418c08bc0
[ 16.456045] Modules linked in: sunrpc nvidia_drm(POE+) nvidia_modeset(POE) nvidia_uvm(OE) intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iwlmvm kvm snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio mac80211 snd_hda_intel irqbypass snd_hda_codec crct10dif_pclmul iTCO_wdt mei_hdcp nvidia(POE) iTCO_vendor_support libarc4 crc32_pclmul snd_hda_core snd_hwdep snd_seq iwlwifi ghash_clmulni_intel intel_cstate drm_kms_helper snd_seq_device intel_uncore snd_pcm i2c_i801 wmi_bmof intel_rapl_perf snd_timer joydev cfg80211 drm jmb38x_ms mei_me snd soundcore rfkill lpc_ich ipmi_devintf memstick mei ipmi_msghandler binfmt_misc ip_tables sdhci_pci firewire_ohci cqhci firewire_core sdhci jme crc32c_intel serio_raw mmc_core crc_itu_t mii wmi video fuse
[ 16.456557] ---[ end trace 7a9ee5a7fc653964 ]---
[ 16.456604] RIP: 0010:nv_drm_get_connector+0xa2/0x1d0 [nvidia_drm]
[ 16.456653] Code: 64 64 d0 ff 48 8b 44 24 20 65 48 33 04 25 28 00 00 00 0f 85 30 01 00 00 48 83 c4 28 4c 89 e8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 <0f> 0b 48 8d 7c 24 10 49 c7 c5 f4 ff ff ff e8 2b 64 d0 ff 49 8b 46
[ 16.456781] RSP: 0018:ffffb58a4056fa78 EFLAGS: 00010297
[ 16.456825] RAX: ffff8ac1f7c80068 RBX: 0000000000000008 RCX: 0000000000000002
[ 16.456871] RDX: ffff8ac1f7c80080 RSI: 0000000000000297 RDI: 0000000000000297
[ 16.456916] RBP: 0000000000000000 R08: ffff8ac21a38ca78 R09: 0000000000000000
[ 16.456962] R10: 000000000000000f R11: ffff8ac1f7f37c80 R12: ffff8ac1f7f38a88
[ 16.457007] R13: ffff8ac1f7c80068 R14: ffff8ac21a38c800 R15: 0000000000000001
[ 16.457053] FS: 00007f6d66bf1940(0000) GS:ffff8ac223380000(0000) knlGS:0000000000000000
[ 16.457146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 16.457193] CR2: 00007f4b20a872d0 CR3: 000000031d5d4005 CR4: 00000000000606e0
As mentioned, all that is needed to reproduce this issue is to run
# dnf install akmod-nvidia
as on the current Fedora releases modesetting is enabled by default. Setting
nvidia-drm.modeset=0
solves the problem for me. This needs to be something specific to this particular configuration as my other computer works fine with modeset enabled.