On my AWS G6 instance with a L4 GPU, running RHEL 9 5.14.0-503.31.1.el9_5.x86_64 and installed the nvidia driver 560.35.03 . nvidia-smi works fine and I noticed that llama.cpp server being slower to respond than a Windows server running the same GPU and instance type in AWS
nvidia-smi
Thu Mar 13 21:09:27 2025
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |
|-----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA L4 Off | 00000000:31:00.0 Off | 0 |
| N/A 32C P0 29W / 72W | 1MiB / 23034MiB | 0% Default |
| | | N/A |
±----------------------------------------±-----------------------±---------------------+
±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
±----------------------------------------------------------------------------------------+
So just to rule out any driver related issues, I uninstalled the nvidia-driver and installed them again using nvidia-driver-local-repo-rhel9-570.124.06-1.0-1.x86_64.rpm (following the NVIDIA driver doc steps for RHEL9).
Now post installation I keep getting “NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver” message when running nvidia-smi.
Anyway to figure out whats going wrong? I have uninstalled/rebooted/re-installed/rebooted this driver a few times without any resolution.