Hi,
I am having trouble with tenserflow not detecting the H100 GPU on my linux ubuntu 22.04 virtual machine. I have installed the nvidia driver 550.120 and tensorflow with cuda in a python 3.11 virtual environment. However when I run
import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))
I get this:
2025-03-05 17:20:42.323506: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1741195242.340859 3706 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1741195242.345548 3706 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-05 17:20:42.362098: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
[]
2025-03-05 17:21:13.623785: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: CUDA_ERROR_SYSTEM_NOT_READY: system not yet initialized
2025-03-05 17:21:13.623843: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:137] retrieving CUDA diagnostic information for host: ps116l7yjshm
2025-03-05 17:21:13.623853: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:144] hostname: ps116l7yjshm
2025-03-05 17:21:13.623992: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:168] libcuda reported version is: 550.120.0
2025-03-05 17:21:13.624014: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:172] kernel reported version is: 550.120.0
2025-03-05 17:21:13.624018: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:259] kernel version seems to match DSO: 550.120.0
my virtual machine has only one H100 GPU, so I dont think it is a fabric manager issue. I have tried for hours to fix this, but I can’t make tensorflow detect the gpu.