Cpp pytorch inference

Gaming and Visualization Technologies APIs OpenGL
, , ,
1

I have updated all drivers, tested with nvidia graphics, updated chrome settings for angle and passthrough and when i try to execute pytorch cpp file, i keep getting

passthrough is not supported gl is disabled cpp inference nvidia graphics

[main 2023-05-30T22:51:36.739Z] Edge term is enabled, starting x2pagentd…
[main 2023-05-30T22:51:36.854Z] update#setState idle
[main 2023-05-30T22:51:37.960Z] [UtilityProcess id: 1, type: extensionHost, pid: ]: creating new…
[main 2023-05-30T22:51:37.978Z] [UtilityProcess id: 1, type: extensionHost, pid: 38184]: successfully created
[main 2023-05-30T22:52:06.855Z] update#setState checking for updates
[main 2023-05-30T22:52:06.963Z] update#setState idle
[39644:0530/155337.113:ERROR:gpu_init.cc(481)] Passthrough is not supported, GL is disabled, ANGLE is

any ideas?

2

Hi there @rossroxas and welcome to the NVIDIA developer forums.

It is really difficult to figure out what kind of issue you are facing. Can you please add a bit more detail?

Start with System specs like HW platform, OS, GPU, Driver and CUDA version, pytorch version.

You are trying to do inference with pytorch. Inference of what? Are you using some NVIDIA SDK?

Also ANGLE is first and foremost an OpenGLES abstraction layer, how does that connect with pytorch inference?

Maybe with additional information someone can help.

Thanks!

3

thanks and sorry about the lack of detail, my brain was mush already.

Wed May 31 04:53:19 2023
±--------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98 Driver Version: 535.98 CUDA Version: 12.2 |
|-----------------------------------------±---------------------±---------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA RTX A5500 WDDM | 00000000:18:00.0 Off | Off |
| 30% 27C P8 5W / 230W | 0MiB / 24564MiB | 0% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+
| 1 NVIDIA RTX A5500 WDDM | 00000000:3B:00.0 Off | Off |
| 30% 29C P8 7W / 230W | 0MiB / 24564MiB | 0% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+
| 2 NVIDIA RTX A5500 WDDM | 00000000:D8:00.0 On | Off |
| 30% 32C P8 15W / 230W | 855MiB / 24564MiB | 1% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+

±--------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 2 N/A N/A 3108 C+G …5n1h2txyewy\ShellExperienceHost.exe N/A |
| 2 N/A N/A 14456 C+G C:\Windows\explorer.exe N/A |
| 2 N/A N/A 14832 C+G …1.0_x64__8wekyb3d8bbwe\Video.UI.exe N/A |
| 2 N/A N/A 16644 C+G …2txyewy\StartMenuExperienceHost.exe N/A |
| 2 N/A N/A 16788 C+G …0_x64__8wekyb3d8bbwe\HxOutlook.exe N/A |
| 2 N/A N/A 17264 C+G …Search_cw5n1h2txyewy\SearchApp.exe N/A |
| 2 N/A N/A 17928 C+G …b3d8bbwe\Microsoft.Media.Player.exe N/A |
| 2 N/A N/A 19708 C+G …CBS_cw5n1h2txyewy\TextInputHost.exe N/A |
| 2 N/A N/A 29920 C+G …_8wekyb3d8bbwe\Microsoft.Photos.exe N/A |
| 2 N/A N/A 32484 C+G …siveControlPanel\SystemSettings.exe N/A |
| 2 N/A N/A 32684 C+G …Search_cw5n1h2txyewy\SearchApp.exe N/A |
| 2 N/A N/A 36888 C+G …t.LockApp_cw5n1h2txyewy\LockApp.exe N/A

I have a windows 10 running
Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz 2.69 GHz (2 processors)
64-bit operating system, x64-based processor

Nvidia specs
535.98-quadro-rtx-desktop-notebook-win10-win11-64bit-international-dch-whql
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\cudnn-windows-x86_64-8.9.0.131_cuda11-archive
i believe running cuda11.8
NVIDIA_Nsight_Perf_SDK_2023.1_Public_Windows

i am using pytorch for model inference and from the pytorch tutorial, using cpp, the file that i am trying to run is torchscript: ts-infer.cpp via vs code, msvs code and i try to execute the command via windows command prompt.

toml 0.10.2
tomli 2.0.1
toolz 0.12.0
torch 2.0.0
torch-model-archiver 0.7.1b20230501
torch-model-archiver-nightly 2023.5.3
torch-package 1.0.1
torch-utils 0.1.2
torch-workflow-archiver 0.2.7b20230430
torch-workflow-archiver-nightly 2023.5.3
torchaudio 2.0.1
torchdata 0.6.0
torchinfo 1.7.2
torchlib 0.1
torchpippy 0.1.0
torchserve 0.7.1
torchserve-nightly 2023.5.3
torchtext 0.15.1
torchutils 0.0.4
torchvision 0.15.1
pytorch-gpu 0.0.1
pytorch-utils 0.5.5

and here

(base) C:\Users\rossroxas>python -m torch.utils.collect_env
C:\miniconda3\lib\site-packages\numpy_distributor_init.py:30: UserWarning: loaded more than 1 DLL from .libs:
C:\miniconda3\lib\site-packages\numpy.libs\libopenblas.QVLO2T66WEPI7JZ63PS3HMOHFEY472BC.gfortran-win_amd64.dll
C:\miniconda3\lib\site-packages\numpy.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
warnings.warn(“loaded more than 1 DLL from .libs:”
Collecting environment information…
PyTorch version: 2.0.0+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Enterprise
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.26.3
Libc version: N/A

Python version: 3.9.16 (main, Mar 8 2023, 10:39:24) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration:
GPU 0: NVIDIA RTX A5500
GPU 1: NVIDIA RTX A5500
GPU 2: NVIDIA RTX A5500

Nvidia driver version: 535.98
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2694
DeviceID=CPU0
Family=179
L2CacheSize=28672
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2694
Name=Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz
ProcessorType=3
Revision=21767

Architecture=9
CurrentClockSpeed=2694
DeviceID=CPU1
Family=179
L2CacheSize=28672
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2694
Name=Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz
ProcessorType=3
Revision=21767

Versions of relevant libraries:
[pip3] libtorch==1.2.0.1
[pip3] mypy==1.2.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.19.3
[pip3] pytorch-gpu==0.0.1
[pip3] pytorch-utils==0.5.5
[pip3] torch==2.0.0
[pip3] torch-model-archiver==0.7.1b20230501
[pip3] torch-model-archiver-nightly==2023.5.3
[pip3] torch-package==1.0.1
[pip3] torch-utils==0.1.2
[pip3] torch-workflow-archiver==0.2.7b20230430
[pip3] torch-workflow-archiver-nightly==2023.5.3
[pip3] torchaudio==2.0.1
[pip3] torchdata==0.6.0
[pip3] torchinfo==1.7.2
[pip3] torchlib==0.1
[pip3] torchpippy==0.1.0
[pip3] torchserve==0.7.1
[pip3] torchserve-nightly==2023.5.3
[pip3] torchtext==0.15.1
[pip3] torchutils==0.0.4
[pip3] torchvision==0.15.1
[conda] blas 1.0 mkl
[conda] cpuonly 1.0 0 pytorch
[conda] cudatoolkit 11.8.0 h09e9e62_11 conda-forge
[conda] libtorch 1.2.0.1 pypi_0 pypi
[conda] mkl 2020.2 256
[conda] mkl-include 2023.1.0 haa95532_46356
[conda] mkl-service 2.3.0 py39h196d8e1_0
[conda] mkl_fft 1.3.0 py39h46781fe_0
[conda] mkl_random 1.0.2 py39h848d8c7_0
[conda] numpy 1.24.0 pypi_0 pypi
[conda] pytorch-gpu 0.0.1 pypi_0 pypi
[conda] pytorch-utils 0.5.5 pypi_0 pypi
[conda] torch 2.0.0 pypi_0 pypi
[conda] torch-model-archiver 0.7.1b20230503 pypi_0 pypi
[conda] torch-model-archiver-nightly 2023.5.3 pypi_0 pypi
[conda] torch-package 1.0.1 pypi_0 pypi
[conda] torch-utils 0.1.2 pypi_0 pypi
[conda] torch-workflow-archiver 0.2.7b20230503 pypi_0 pypi
[conda] torch-workflow-archiver-nightly 2023.5.3 pypi_0 pypi
[conda] torchaudio 2.0.1 pypi_0 pypi
[conda] torchdata 0.6.0 pypi_0 pypi
[conda] torchinfo 1.7.2 pyhd8ed1ab_0 conda-forge
[conda] torchlib 0.1 pypi_0 pypi
[conda] torchpippy 0.1.0 pypi_0 pypi
[conda] torchserve 0.7.1 pypi_0 pypi
[conda] torchserve-nightly 2023.5.3 pypi_0 pypi
[conda] torchtext 0.15.1 pypi_0 pypi
[conda] torchutils 0.0.4 pypi_0 pypi
[conda] torchvision 0.15.1 pypi_0 pypi

let me know if i missed anything!

thanks!

4

The above does not bode well to be honest. I suspect something in your driver installation went wrong. PyTorch sees your GPUs and driver, but cannot access CUDA.

Do you recall how you installed the CUDA toolkit? I think PyTorch recommends using the package specific installation methods as part of PyTorch installation. That way it will be in the correct PATH environment of PyTorch.

You might want to check that and re-install.

If that does not help, we can look at other options.

Thanks!

5

this is what i had in april

Python version: 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA RTX A5500
GPU 1: NVIDIA RTX A5500
GPU 2: NVIDIA RTX A5500

Nvidia driver version: 522.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2693
DeviceID=CPU0
Family=179
L2CacheSize=28672
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2693
Name=Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz
ProcessorType=3
Revision=21767

Architecture=9
CurrentClockSpeed=2693
DeviceID=CPU1
Family=179
L2CacheSize=28672
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2693
Name=Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz
ProcessorType=3
Revision=21767

Versions of relevant libraries:
[pip3] numpy==1.16.6
[pip3] torch==2.0.0+cu118
[pip3] torchaudio==2.0.1+cu118
[pip3] torchvision==0.15.1+cu118
[conda] cudatoolkit 11.8.0 h09e9e62_11 conda-forge
[conda] mkl 2023.1.0 h8bd8f75_46356
[conda] mkl-include 2023.1.0 haa95532_46356
[conda] numpy 1.23.5 pypi_0 pypi
[conda] pytorch-gpu 0.0.1 pypi_0 pypi
[conda] torch 2.0.0 pypi_0 pypi
[conda] torch-package 1.0.1 pypi_0 pypi
[conda] torch-utils 0.1.2 pypi_0 pypi
[conda] torchlib 0.1 pypi_0 pypi
[conda] torchutils 0.0.4 pypi_0 pypi
[conda] torchvision 0.15.1 pypi_0 pypi

I had updated from 11.8 → 12.1 and now the current 12.2
should i have stuck with 11.8?

I was getting the same output with 11.7 and 11.8:
passthrough and GL issues. I was testing out the updated Nsight graphics UI, hoping it would solve my problem, still learning it.

is it now a test and see or is there a specific version i should use with what I have and what i doing.

GOAL: deploy model saved as .cpp

6

My recommendation would be to follow the PyTorch instructions on which CUDA package to install and how. If they support 12.2 then fine, but if not better stay with 11.8.

On the GL pass-through part I can’t help right away. Did it interfere with the inference or cause other problems? Otherwise you might just ignore it as a warning message.

7

yes, i started to uninstall and try to keep things are they were before to avoid more conflicts.

i just need to run the model through inference but keep running into the error on msvs code and vs code and was hoping to get some breadcrumbs to lead me to that point.

8

Here is my environment

this is my laptop, my windows desktop shows the same after i cleaned up some things.

I am followed the steps from pytorch and for inference with my model, I added and changed a few things that came about from errors i learned about. due to the passthrough being disabled, is that something that can be ignored? I went digging a bit an found that I need a VM to access and make changes to the GPU so that it can be interoperable with memory. is there nothing in the nvidia-smi or nvidia configureDriver that can be updated?

thanks!

ls.collect_env
Collecting environment information…
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Enterprise
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:45) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3070 Ti Laptop GPU
Nvidia driver version: 536.23
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2700
DeviceID=CPU0
Family=198
L2CacheSize=7680
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2700
Name=12th Gen Intel(R) Core™ i7-12700H
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] lbs-pytorch==1.1.20220705
[pip3] numpy==1.22.3
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] torchvision==0.15.2
[conda] blas 2.114 mkl conda-forge
[conda] blas-devel 3.9.0 14_win64_mkl conda-forge
[conda] cudatoolkit 11.3.1 h59b6b97_2
[conda] lbs-pytorch 1.1.20220705 pypi_0 pypi
[conda] libblas 3.9.0 14_win64_mkl conda-forge
[conda] libcblas 3.9.0 14_win64_mkl conda-forge
[conda] liblapack 3.9.0 14_win64_mkl conda-forge
[conda] liblapacke 3.9.0 14_win64_mkl conda-forge
[conda] mkl 2022.0.0 h0e2418a_796 conda-forge
[conda] mkl-devel 2022.0.0 h57928b3_797 conda-forge
[conda] mkl-include 2022.0.0 h0e2418a_796 conda-forge
[conda] numpy 1.22.3 py38h5ed9b9d_2 conda-forge
[conda] pytorch 2.0.1 py3.8_cuda11.8_cudnn8_0 pytorch
[conda] pytorch-cuda 11.8 h24eeafa_5 pytorch
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torch 2.0.1 pypi_0 pypi
[conda] torchaudio 2.0.2 pypi_0 pypi
[conda] torchvision 0.15.2 pypi_0 pypi

9

One thing that still seems to be a mismatch in your setup is the fact that you use conda-forge. Recommedned channels are nvidia, main or pytorch. from one of our engineers I received a recommended installation command:

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

But make sure to start with a clean conda environment!

Beyond that there is not much we can do, since NVIDIA does not build their own Windows based PyTorch/CUDA integration.