Failed to initialize NVML: Driver/library version mismatch NVML library version: 535.171

Bug report:
nvidia-bug-report.log (1.6 MB)

i am facing a version mismatch and cannot use gpu because i am getting this error frequently.
help me solve it please.

this is what i saw in bug report

2024-04-15T07:36:02.625258+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1069.050090] NVRM: API mismatch: the client has the version 535.171.04, but
2024-04-15T07:36:02.625287+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1069.050090] NVRM: this kernel module has the version 535.161.07. Please
2024-04-15T07:36:02.625291+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1069.050090] NVRM: make sure that this kernel module and all NVIDIA driver
2024-04-15T07:36:02.625293+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1069.050090] NVRM: components have the same version.
2024-04-15T07:38:16.917272+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1203.340948] NVRM: API mismatch: the client has the version 535.171.04, but
2024-04-15T07:38:16.917297+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1203.340948] NVRM: this kernel module has the version 535.161.07. Please
2024-04-15T07:38:16.917302+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1203.340948] NVRM: make sure that this kernel module and all NVIDIA driver
2024-04-15T07:38:16.917305+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1203.340948] NVRM: components have the same version.
2024-04-15T07:44:17.769273+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1564.193106] NVRM: API mismatch: the client has the version 535.171.04, but
2024-04-15T07:44:17.769298+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1564.193106] NVRM: this kernel module has the version 535.161.07. Please
2024-04-15T07:44:17.769302+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1564.193106] NVRM: make sure that this kernel module and all NVIDIA driver
2024-04-15T07:44:17.769305+05:45 rohan-manandhar-Nitro-AN515-44 kernel: [ 1564.193106] NVRM: components have the same version.

Looks like you updated the driver and didn’t reboot.

Facing same problem with exactly these driver versions on my server today. Didin’t update drivers. Looks like it updated yesterday automatically. Going to do this (How to prevent NVIDIA drivers from automatically upgrading on Ubuntu? - Stack Overflow) to prevent this in future.

Yeah i was shocked too, it was automatically updated and the mismatch was fixed. BTW thanks for the stack cheers.