RmInitAdapter failed

[Fri Mar 6 15:35:38 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:35:38 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:35:43 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:35:43 2020] NVRM: rm_init_adapter failed for device bearing minor number 1
[Fri Mar 6 15:36:38 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:36:38 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:36:43 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:36:43 2020] NVRM: rm_init_adapter failed for device bearing minor number 1
[Fri Mar 6 15:37:37 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:37:37 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:37:41 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:37:41 2020] NVRM: rm_init_adapter failed for device bearing minor number 1
[Fri Mar 6 15:37:45 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:37:45 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:37:50 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:37:50 2020] NVRM: rm_init_adapter failed for device bearing minor number 1
[Fri Mar 6 15:37:54 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:37:54 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:37:58 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:37:58 2020] NVRM: rm_init_adapter failed for device bearing minor number 1
[Fri Mar 6 15:38:03 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:38:03 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:38:07 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:38:07 2020] NVRM: rm_init_adapter failed for device bearing minor number 1
[Fri Mar 6 15:38:38 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:38:38 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:38:43 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:38:43 2020] NVRM: rm_init_adapter failed for device bearing minor number 1
[Fri Mar 6 15:39:38 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:39:38 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:39:43 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:39:43 2020] NVRM: rm_init_adapter failed for device bearing minor number 1
[Fri Mar 6 15:40:35 2020] NVRM: RmInitAdapter failed! (0x25:0x51:1084)
[Fri Mar 6 15:40:35 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:40:39 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:40:39 2020] NVRM: rm_init_adapter failed for device bearing minor number 1
[Fri Mar 6 15:41:38 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:41:38 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:41:43 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:41:43 2020] NVRM: rm_init_adapter failed for device bearing minor number 1
[Fri Mar 6 15:42:38 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:42:38 2020] NVRM: rm_init_adapter failed for device bearing minor number 0
[Fri Mar 6 15:42:43 2020] NVRM: RmInitAdapter failed! (0x26:0xffff:1106)
[Fri Mar 6 15:42:43 2020] NVRM: rm_init_adapter failed for device bearing minor number 1

====

op@hadoop-0006:~$ nvidia-smi
Fri Mar 6 15:40:26 2020
±----------------------------------------------------------------------------+
| NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA Version: 10.1 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108… On | 00000000:82:00.0 Off | N/A |
| 64% 78C P2 74W / 250W | 8747MiB / 11178MiB | 2% Default |
±------------------------------±---------------------±---------------------+
| 1 GeForce GTX 108… On | 00000000:83:00.0 Off | N/A |
| 63% 78C P2 246W / 250W | 8743MiB / 11178MiB | 100% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 19244 C python 2186MiB |
| 0 19246 C python 2186MiB |
| 0 19247 C python 2176MiB |
| 0 19250 C python 2186MiB |
| 1 21974 C python 2202MiB |
| 1 21980 C python 2176MiB |
| 1 21983 C python 2166MiB |
| 1 21986 C python 2186MiB |
±----------------------------------------------------------------------------+
nvidia-bug-report.log.gz (1.81 MB)

Please enable the nvidia-persistenced to start on boot, make sure it is continuously running and check if that resolves the issue.