RuntimeError: Error code: 98, reason: invalid device function

[rakhi.proeffico]

Hello there,
I am trying to run face_recognition code using OpenCV with CUDA support.
The code I am running is;

for name in images:
    image = face_recognition.load_image_file(name)
    encoding = face_recognition.face_encodings(image)[0]
    known_faces.append(encoding)
    known_names.append(name)

complete error that i get is;

RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_4348/254179735.py in <module>
      8 for name in images:
      9     image = face_recognition.load_image_file(name)
---> 10     encoding = face_recognition.face_encodings(image)[0]
     11     known_faces.append(encoding)
     12     known_names.append(name)

~/.local/lib/python3.10/site-packages/face_recognition/api.py in face_encodings(face_image, known_face_locations, num_jitters, model)
    212     """
    213     raw_landmarks = _raw_face_landmarks(face_image, known_face_locations, model)
--> 214     return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landmark_set, num_jitters)) for raw_landmark_set in raw_landmarks]
    215 
    216 

~/.local/lib/python3.10/site-packages/face_recognition/api.py in <listcomp>(.0)
    212     """
    213     raw_landmarks = _raw_face_landmarks(face_image, known_face_locations, model)
--> 214     return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landmark_set, num_jitters)) for raw_landmark_set in raw_landmarks]
    215 
    216 

RuntimeError: Error while calling cudaOccupancyMaxPotentialBlockSize(&num_blocks,&num_threads,K) in file /tmp/pip-install-tfazu_bm/dlib_65f79a31ba2f4549a39bdef8017ea1ef/dlib/cuda/cuda_utils.h:186. code: 98, reason: invalid device function

I am having NVIDIA GeForce RTX 3060 graphics card with NVIDIA-SMI 525.89.02, CUDA Version 12.0 with cuDNN on

I need help on this

[Robert_Crovella]

The usual meaning of this error is that something you are running is not built correctly to run on your GPU.

[rakhi.proeffico]

How to fix that?? What should be done now ?!

[striker159]

is RTX 3060 supported by the library you want to use?

[Agentirish]

@rakhi.proeffico The error you’re encountering seems to be related to an incompatibility between the CUDA version and the Dlib library compiled with CUDA support. To resolve this issue, you can follow these steps:

1. Uninstall the current Dlib installation:

pip uninstall dlib

2. Install the required dependencies for building Dlib:

sudo apt-get install build-essential cmake libopenblas-dev liblapack-dev libx11-dev libgtk-3-dev python3-dev python3-numpy

3. Clone the Dlib repository:

git clone GitHub - davisking/dlib: A toolkit for making real world machine learning and data analysis applications in C++

4. Change to the Dlib directory:

cd dlib

5. Make sure that you have the correct CUDA version in your environment. For example, you can set the environment variable CUDA_HOME to the CUDA installation directory:

export CUDA_HOME=/usr/local/cuda-12.0
Note: Replace /usr/local/cuda-12.0 with your actual CUDA installation directory.

6. Build Dlib with CUDA support:

mkdir build
cd build
cmake -DUSE_AVX_INSTRUCTIONS=1 -D DLIB_USE_CUDA=1 …
cmake --build .

7. Build and install the Python wheel:

cd …
python setup.py install --set DLIB_USE_CUDA=1 --set USE_AVX_INSTRUCTIONS=1

8. Verify the Dlib installation:

python -c “import dlib; print(dlib.DLIB_USE_CUDA)”

---

This command should output True , which indicates that Dlib is using CUDA.

Now, try running your face_recognition code again. The error should be resolved. If you still encounter any issues, make sure your graphics card drivers, CUDA toolkit, and cuDNN library are properly installed and compatible with each other.

[rakhi.proeffico]

CUDA was found but your compiler failed to compile a simple CUDA program so dlib isn't going to use CUDA.
The output of the failed CUDA test compile is shown below: 
-- *** 
-- ***   Change Dir: /home/proeffico/dlib/build/temp.linux-x86_64-3.10/dlib_build/cuda_test_build
   ***   
   ***   Run Build Command(s):/usr/bin/gmake -f Makefile && [ 50%] Building NVCC (Device) object CMakeFiles/cuda_test.dir/cuda_test_generated_cuda_test.cu.o
   ***   nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
   ***   nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
   ***   /usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
   ***     435 |         function(_Functor&& __f)
   ***         |

This is the error i have got while building dlib.
And output to import dlib; print(dlib.DLIB_USE_CUDA) is False

[Agentirish]

Looks like there is a incompatibility between the CUDA version and the C++ version you are using.
Try these steps and let me know if you got it working…

1. Update your CUDA version: Make sure you have the latest CUDA version installed that is compatible with your GPU. You can check the compatibility and download the latest version from the NVIDIA website: https://developer.nvidia.com/cuda-downloads

2. Downgrade your GCC version: There might be an incompatibility between the C++ version (GCC) and the CUDA version. Try downgrading GCC to a version that is compatible with the CUDA version you have installed. You can check the compatibility from the CUDA installation guide: NVIDIA Documentation

To downgrade GCC, you can use the following commands:

sudo apt-get install gcc-8 g+±8
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 50 --slave /usr/bin/g++ g++ /usr/bin/g+±8

3. Rebuild dlib with CUDA support:

After updating/downgrading the necessary components, try rebuilding dlib with CUDA support.
First, clean the previous build:

rm -rf dlib/build

Then, rebuild dlib:

cd dlib
mkdir build
cd build
cmake …
cmake --build . --config Release
sudo make install
sudo ldconfig

4. Test if dlib is using CUDA:

Run the following Python code to check if dlib is now using CUDA:

import dlib
print(dlib.DLIB_USE_CUDA)

If everything is set up correctly, the output should be TRUE

Hope this helps!