After playing with nouveau for a while I decided to go back to the binary driver - mainly due to vulkan support. Unfortunately the issue is still there with Fedora 31, 435.21 driver and 5.3.7 kernel:

[   16.454126] ------------[ cut here ]------------
[   16.454127] kernel BUG at /tmp/akmodsbuild.XCdjp5Qz/BUILD/nvidia-kmod-435.21/_kmod_build_5.3.7-301.fc31.x86_64/nvidia-drm/nvidia-drm-connector.c:439!
[   16.454144] invalid opcode: 0000 [#1] SMP PTI
[   16.454157] CPU: 6 PID: 736 Comm: systemd-udevd Tainted: P           OE     5.3.7-301.fc31.x86_64 #1
[   16.454170] Hardware name: CLEVO                             P150HMx/P150HMx, BIOS 4.6.4 08/09/2011
[   16.454188] RIP: 0010:nv_drm_get_connector+0xa2/0x1d0 [nvidia_drm]
[   16.454199] Code: 64 64 d0 ff 48 8b 44 24 20 65 48 33 04 25 28 00 00 00 0f 85 30 01 00 00 48 83 c4 28 4c 89 e8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 <0f> 0b 48 8d 7c 24 10 49 c7 c5 f4 ff ff ff e8 2b 64 d0 ff 49 8b 46
[   16.454259] RSP: 0018:ffffb58a4056fa78 EFLAGS: 00010297
[   16.454306] RAX: ffff8ac1f7c80068 RBX: 0000000000000008 RCX: 0000000000000002
[   16.454354] RDX: ffff8ac1f7c80080 RSI: 0000000000000297 RDI: 0000000000000297
[   16.454402] RBP: 0000000000000000 R08: ffff8ac21a38ca78 R09: 0000000000000000
[   16.454451] R10: 000000000000000f R11: ffff8ac1f7f37c80 R12: ffff8ac1f7f38a88
[   16.454497] R13: ffff8ac1f7c80068 R14: ffff8ac21a38c800 R15: 0000000000000001
[   16.454544] FS:  00007f6d66bf1940(0000) GS:ffff8ac223380000(0000) knlGS:0000000000000000
[   16.454626] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   16.454677] CR2: 00007f4b20a872d0 CR3: 000000031d5d4005 CR4: 00000000000606e0
[   16.454726] Call Trace:
[   16.454776]  nv_drm_add_encoder+0x19d/0x300 [nvidia_drm]
[   16.454823]  nv_drm_load+0x264/0x307 [nvidia_drm]
[   16.454869]  ? nv_drm_master_drop+0x60/0x60 [nvidia_drm]
[   16.454927]  drm_dev_register+0x111/0x150 [drm]
[   16.454977]  nv_drm_probe_devices+0x196/0x220 [nvidia_drm]
[   16.455024]  ? 0xffffffffc04e2000
[   16.455073]  do_one_initcall+0x46/0x1f4
[   16.455119]  ? _cond_resched+0x15/0x30
[   16.455164]  ? kmem_cache_alloc_trace+0x162/0x220
[   16.455210]  ? do_init_module+0x23/0x230
[   16.455254]  do_init_module+0x5c/0x230
[   16.455302]  load_module+0x27b1/0x2990
[   16.455350]  ? __do_sys_finit_module+0xaa/0x110
[   16.455396]  __do_sys_finit_module+0xaa/0x110
[   16.455448]  do_syscall_64+0x5f/0x1a0
[   16.455497]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   16.455544] RIP: 0033:0x7f6d67c4115d
[   16.455590] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d fb 5c 0c 00 f7 d8 64 89 01 48
[   16.455718] RSP: 002b:00007fff90ac23a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[   16.455805] RAX: ffffffffffffffda RBX: 0000565418c08bc0 RCX: 00007f6d67c4115d
[   16.455853] RDX: 0000000000000000 RSI: 0000565418c12b90 RDI: 0000000000000017
[   16.455902] RBP: 0000000000020000 R08: 0000000000000000 R09: 0000000000000007
[   16.455949] R10: 0000000000000017 R11: 0000000000000246 R12: 0000565418c12b90
[   16.455997] R13: 0000000000000000 R14: 000056541893f290 R15: 0000565418c08bc0
[   16.456045] Modules linked in: sunrpc nvidia_drm(POE+) nvidia_modeset(POE) nvidia_uvm(OE) intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iwlmvm kvm snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio mac80211 snd_hda_intel irqbypass snd_hda_codec crct10dif_pclmul iTCO_wdt mei_hdcp nvidia(POE) iTCO_vendor_support libarc4 crc32_pclmul snd_hda_core snd_hwdep snd_seq iwlwifi ghash_clmulni_intel intel_cstate drm_kms_helper snd_seq_device intel_uncore snd_pcm i2c_i801 wmi_bmof intel_rapl_perf snd_timer joydev cfg80211 drm jmb38x_ms mei_me snd soundcore rfkill lpc_ich ipmi_devintf memstick mei ipmi_msghandler binfmt_misc ip_tables sdhci_pci firewire_ohci cqhci firewire_core sdhci jme crc32c_intel serio_raw mmc_core crc_itu_t mii wmi video fuse
[   16.456557] ---[ end trace 7a9ee5a7fc653964 ]---
[   16.456604] RIP: 0010:nv_drm_get_connector+0xa2/0x1d0 [nvidia_drm]
[   16.456653] Code: 64 64 d0 ff 48 8b 44 24 20 65 48 33 04 25 28 00 00 00 0f 85 30 01 00 00 48 83 c4 28 4c 89 e8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 <0f> 0b 48 8d 7c 24 10 49 c7 c5 f4 ff ff ff e8 2b 64 d0 ff 49 8b 46
[   16.456781] RSP: 0018:ffffb58a4056fa78 EFLAGS: 00010297
[   16.456825] RAX: ffff8ac1f7c80068 RBX: 0000000000000008 RCX: 0000000000000002
[   16.456871] RDX: ffff8ac1f7c80080 RSI: 0000000000000297 RDI: 0000000000000297
[   16.456916] RBP: 0000000000000000 R08: ffff8ac21a38ca78 R09: 0000000000000000
[   16.456962] R10: 000000000000000f R11: ffff8ac1f7f37c80 R12: ffff8ac1f7f38a88
[   16.457007] R13: ffff8ac1f7c80068 R14: ffff8ac21a38c800 R15: 0000000000000001
[   16.457053] FS:  00007f6d66bf1940(0000) GS:ffff8ac223380000(0000) knlGS:0000000000000000
[   16.457146] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   16.457193] CR2: 00007f4b20a872d0 CR3: 000000031d5d4005 CR4: 00000000000606e0

As mentioned, all that is needed to reproduce this issue is to run

# dnf install akmod-nvidia

as on the current Fedora releases modesetting is enabled by default. Setting

nvidia-drm.modeset=0

solves the problem for me. This needs to be something specific to this particular configuration as my other computer works fine with modeset enabled.