How to role back drivers from nVidia-smi version 410.79 down to 396.44

I work at a financial securities company where our trading software is particular on not only the OS version but the graphic card driver version as well.

We currently run Red Hat Enterprise 7.5, and nVidia-smi version 410.79… According to the trading software manufacturer, they require us to run nVidia-smi version 396.44. What is the best way do you recommend we rollback to this lower version?

I tried uninstalling cuda-driver-410.79 and installing cuda-driver-396.44… it looks like there are other drivers installed that need change that I’m not sure which are required to change.

Please assist thanks!

Joe

How did you install the driver/cuda? Which cuda version are you using?

Hi,

Here are the list of cuda software installed on this server:

[root@server joe]# rpm -qa|grep cuda
cuda-tools-10-0-10.0.130-1.x86_64
cuda-10.0.130-1.x86_64
cuda-nvdisasm-10-0-10.0.130-1.x86_64
cuda-curand-dev-10-0-10.0.130-1.x86_64
cuda-cublas-10-0-10.0.130-1.x86_64
cuda-misc-headers-10-0-10.0.130-1.x86_64
cuda-libraries-10-0-10.0.130-1.x86_64
cuda-gpu-library-advisor-10-0-10.0.130-1.x86_64
cuda-compiler-10-0-10.0.130-1.x86_64
nvidia-driver-cuda-410.79-1.el7.x86_64
cuda-driver-dev-10-0-10.0.130-1.x86_64
cuda-cuobjdump-10-0-10.0.130-1.x86_64
cuda-cusolver-dev-10-0-10.0.130-1.x86_64
cuda-npp-dev-10-0-10.0.130-1.x86_64
cuda-samples-10-0-10.0.130-1.x86_64
cuda-nvvp-10-0-10.0.130-1.x86_64
cuda-demo-suite-10-0-10.0.130-1.x86_64
cuda-cudart-dev-10-0-10.0.130-1.x86_64
cuda-nvgraph-10-0-10.0.130-1.x86_64
cuda-nvrtc-10-0-10.0.130-1.x86_64
cuda-nvjpeg-10-0-10.0.130-1.x86_64
cuda-cusparse-10-0-10.0.130-1.x86_64
cuda-nvprof-10-0-10.0.130-1.x86_64
cuda-memcheck-10-0-10.0.130-1.x86_64
cuda-command-line-tools-10-0-10.0.130-1.x86_64
cuda-documentation-10-0-10.0.130-1.x86_64
cuda-nsight-10-0-10.0.130-1.x86_64
cuda-toolkit-10-0-10.0.130-1.x86_64
cuda-runtime-10-0-10.0.130-1.x86_64
nvidia-driver-cuda-libs-410.79-1.el7.x86_64
cuda-curand-10-0-10.0.130-1.x86_64
cuda-cufft-10-0-10.0.130-1.x86_64
cuda-nvgraph-dev-10-0-10.0.130-1.x86_64
cuda-nvrtc-dev-10-0-10.0.130-1.x86_64
cuda-nvjpeg-dev-10-0-10.0.130-1.x86_64
cuda-cusparse-dev-10-0-10.0.130-1.x86_64
cuda-libraries-dev-10-0-10.0.130-1.x86_64
cuda-nvprune-10-0-10.0.130-1.x86_64
cuda-nvcc-10-0-10.0.130-1.x86_64
cuda-drivers-410.79-1.x86_64
cuda-license-10-0-10.0.130-1.x86_64
cuda-cufft-dev-10-0-10.0.130-1.x86_64
cuda-cusolver-10-0-10.0.130-1.x86_64
cuda-npp-10-0-10.0.130-1.x86_64
cuda-nvtx-10-0-10.0.130-1.x86_64
cuda-visual-tools-10-0-10.0.130-1.x86_64
cuda-10-0-10.0.130-1.x86_64
cuda-cudart-10-0-10.0.130-1.x86_64
cuda-cublas-dev-10-0-10.0.130-1.x86_64
cuda-nvml-dev-10-0-10.0.130-1.x86_64
cuda-gdb-10-0-10.0.130-1.x86_64
cuda-nsight-compute-10-0-10.0.130-1.x86_64
cuda-cupti-10-0-10.0.130-1.x86_64

How do you suggest I rollback the driver version below from 410.79 down to 396.44?

[root@sazemxsglnint08 josephmathieu]# nvidia-smi
Tue May 21 16:50:23 2019
±----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79 Driver Version: 410.79 CUDA Version: 10.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE… Off | FCCC9E73:00:00.0 Off | Off |
| N/A 32C P0 41W / 250W | 466MiB / 16130MiB | 12% Default |
±------------------------------±---------------------±---------------------+
| 1 Tesla V100-PCIE… Off | FCCCAE2F:00:00.0 Off | Off |
| N/A 28C P0 25W / 250W | 11MiB / 16130MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 29463 C ./mpcstub 453MiB |
±----------------------------------------------------------------------------+

Please let me know thanks!

Joe

You have cuda 10 installed, that doesn’t work with a v396 driver (cuda 9.2 max) so you can simply purge cuda and the driver:
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#removing-cuda-tk-and-driver
Afterwards, download the 396.44 driver for RHEL:
https://www.nvidia.com/download/driverResults.aspx/136948/en-us
and install.
Then you can download the cuda 9.2 rpm from archives:
https://developer.nvidia.com/cuda-92-download-archive?target_os=Linux&target_arch=x86_64&target_distro=RHEL&target_version=7&target_type=rpmlocal
and don’t install cuda, instead install cuda-toolkit-9-2 otherwise you would overwrite the driver:

sudo rpm -i cuda-repo-rhel7-9-2-local-9.2.148-1.x86_64.rpm
sudo yum clean all
sudo yum install cuda-toolkit-9-2

I’ll give this a shot tomorrow… I’ll provide you an update when done.

Thank you very much for your help!

Hi Generix…

So I removed all the cuda related rpms and installed the two rpms you instructed to install. Now I tried to run the nvidia-smi command and the file is not available. Is there another step I need to do?

Wooh-hoo!!!.. it worked!

I redid your steps and yes there’s a few more things I found from another page that helped. The last main step was a reboot…

All is well now.

Take care!