X does not start due to nvidia_drm module

Hello,
I am facing a problem with startX and GTX 1650 driver on my laptop since updating many packages a few weeks back.
My card is detected by the nvidia driver, and CUDA is working.
But if I do startx, X will segfault, abort or hang depending on the version of the driver ( I have tried the distro, 3rd party and nVidia drivers).
If I remove the nvidia-drm module
sudo rmmod nvidia-drm, then startx will work.
However, nvidia-settings shows a blank.

ERROR: Unable to load info from any available system
(nvidia-settings:3121): GLib-GObject-CRITICAL **: 18:21:13.466: g_object_unref: assertion ‘G_IS_OBJECT (object)’ failed

I have Acer Aspire 7,
AMD Ryzen 5 5500U with Radeon Graphics
TU117M [GeForce GTX 1650 Mobile / Max-Q]
Ubuntu 20.04.2 LTS
Kernel 5.4.0-74-generic

cuda-drivers-465/unknown,now 465.19.01-1 amd64 [installed,automatic]
libnvidia-cfg1-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
libnvidia-common-465/unknown,now 465.19.01-0ubuntu1 all [installed,automatic]
libnvidia-compute-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
libnvidia-decode-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
libnvidia-encode-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
libnvidia-extra-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
libnvidia-fbc1-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
libnvidia-gl-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
libnvidia-ifr1-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
linux-modules-nvidia-465-5.4.0-74-generic/focal-updates,focal-security,now 5.4.0-74.83+1 amd64 [installed]
linux-objects-nvidia-465-5.4.0-74-generic/focal-updates,focal-security,now 5.4.0-74.83+1 amd64 [installed,automatic]
nvidia-compute-utils-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
nvidia-dkms-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
nvidia-driver-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed]
nvidia-kernel-common-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed]
nvidia-kernel-source-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
nvidia-utils-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]
xserver-xorg-video-nvidia-465/focal-updates,focal-security,now 465.27-0ubuntu0.20.04.2 amd64 [installed,automatic]

~$ lshw -c video
*-display
description: 3D controller
product: TU117M [GeForce GTX 1650 Mobile / Max-Q]
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:01:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: bus_master cap_list
configuration: driver=nvidia latency=0
resources: iomemory:fc0-fbf iomemory:fc0-fbf irq:74 memory:d0000000-d0ffffff memory:fcc0000000-fccfffffff memory:fcd0000000-fcd1ffffff ioport:3000(size=128)
*-display UNCLAIMED
description: VGA compatible controller
product: Lucienne
vendor: Advanced Micro Devices, Inc. [AMD/ATI]
physical id: 0
bus info: pci@0000:05:00.0
version: c2
width: 64 bits
clock: 33MHz
capabilities: vga_controller bus_master cap_list
configuration: latency=0
resources: iomemory:fc0-fbf iomemory:fc0-fbf memory:fce0000000-fcefffffff memory:fcf0000000-fcf01fffff ioport:1000(size=256) memory:d1400000-d147ffff

nvidia-smi
Sun Jun 20 18:17:20 2021
±----------------------------------------------------------------------------+
| NVIDIA-SMI 465.27 Driver Version: 465.27 CUDA Version: 11.3 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce … Off | 00000000:01:00.0 Off | N/A |
| N/A 50C P0 21W / N/A | 56MiB / 3911MiB | 100% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3097 C ./hello 53MiB |
±----------------------------------------------------------------------------+

Thanks.