Error running MLPerf benchmark with CUDA cuInit failed: initialization error

Accelerated Computing CUDA CUDA Setup and Installation
, ,
1

I’m trying to run the MLPerf benchmark with the following command:

sudo make run RUN_ARGS="--benchmarks=bert --scenarios=offline --config_ver=default,high_accuracy,triton,high_accuracy_triton"

However, I’m encountering the following error:

pycuda._driver.LogicError: cuInit failed: initialization error

Environment:

  • NVIDIA-SMI: 550.54.15
  • CUDA Version: 12.4
  • GPU: NVIDIA A100 80GB PCIe
  • CUDA compiler version: 12.0

Additional information:

  • I have verified that the NVIDIA driver and CUDA are properly installed.
  • The GPU is detected correctly by nvidia-smi.
  • nvcc --version shows the CUDA compiler version.

image
image944×725 164 KB

I’m unsure why cuInit is failing. Any insights or suggestions would be greatly appreciated. Thank you!

2

You have MIG mode enabled but no MIG devices defined. That results in a GPU that is unusable.