2nd GPU not showing in nvidia-smi in Ubuntu 22.04

[anjum.sayed48]

I’ve found these two similar threads:

NVRM: GPU 0000:01:00.0: RmInitAdapter failed! - driver 530.30.02

Hello, I have an AMD host system with one H100 on processor 1 and two A100s on processor 2. After some random interval of time measured in days or a few weeks, one and eventually both A100s will vanish from nvidia-smi and CUDA API visibility and the system dmesg log will fill up with [865094.718348] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0xffff:2356) [865094.720365] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0 repeating every few seconds. I’ve tried eve…

1 of 6 gpu is disapeared in nvidia-smi and still shows in lspci command in ubuntu 22.04 and cuda 12.1

Hi, I’m new in installation. We first successfully installed cuda 12.1 on ubuntu 22.04 for 6 A800 GPUs. After cuda installed, nvidia-smi shows 6 GPUs but after few samples run, it shows only 5 GPUs. The 2nd one is missing. We tried to reinstalled cuda and it shows 6 again. And then we run some samples, and it shows 5 GPUs again but this time, the 3rd GPU is missing. And following logs shows this : nvidia-smi: ±-------------------------------------------------------------------------…