We installed a new gpu rtxtitan on our server.then we reinstalled the driver and cuda.nvidia-smi could show the information of new gpu.but we can not train our model on the new gpu.our Operating System is centos7.
| NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN Xp Off | 00000000:02:00.0 Off | N/A |
| 40% 70C P2 235W / 250W | 11763MiB / 12195MiB | 99% Default |
±------------------------------±---------------------±---------------------+
| 1 TITAN RTX Off | 00000000:03:00.0 Off | N/A |
| 42% 60C P0 66W / 280W | 0MiB / 24190MiB | 0% Default |
when we use CUDA_VISIBLE_DEVICES=1 but it works on GPU 0;