How to downgrade nvidia driver

Our server use nvidia driver 535 previously, but I want to downgrade it. I purged driver using apt, and reinstall one in version 470. But then run nvidia-smi gives me:
Failed to initialize NVML: Driver/library version mismatch.
And dmesg:
[ 243.249000] rfkill: input handler disabled
[ 243.697920] NVRM: API mismatch: the client has the version 515.43.04, but
NVRM: this kernel module has the version 470.239.06. Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
[ 1077.038346] NVRM: API mismatch: the client has the version 515.43.04, but
NVRM: this kernel module has the version 470.239.06. Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
[ 1389.116438] NVRM: API mismatch: the client has the version 515.43.04, but
NVRM: this kernel module has the version 470.239.06. Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
[ 1451.451324] NVRM: API mismatch: the client has the version 515.43.04, but
NVRM: this kernel module has the version 470.239.06. Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
It seems that higher version 515 is still on my machine, how can I get rid of it and use version 470?