Unable to load nvidia peermem module

I have installed nvidia driver and cuda toolkit.


i can see that nvidia-peermem module is present.

but i am not able to load it. i see following error:

dmesg doesnt show any error related to peermem after executing above command:

old dmesg logs show following:
image

my roce nic drivers are installed and i am able to see my roce devices:

any idea what is happening and how can i resolve it?
Can anyone please help here?

Thanks in advance!