Onnx runtime GPU

[tcemadhavan]

Hi, i have jetpack 6.2 installed and i’m trying to install onnxruntime-gpu.
First i downloaded onnxruntime using this command.
“pip install -U onnxruntime”
and downloaded the onnxruntime-gpu file using “jp6/cu126 index” this link.

and i tried to check the availability. but i’m getting only ‘AzureExecutionProvider’ and ‘CPUExecutionProvider’. Cuda is not coming.

Did i miss something or do i have to do something else?

[AastaLLL]

Hi,

As your onnxruntime has AzureExecutionProvider, the package should come from a third party instead of our apt server (the one installed from the pip?).
Could you try the command below? We have confirmed that CUDAExecutionProvider exists in the ort.get_available_providers().

$ wget https://pypi.jetson-ai-lab.dev/jp6/cu126/+f/f6e/2baa664069470/onnxruntime_gpu-1.20.2-cp310-cp310-linux_aarch64.whl#sha256=f6e2baa664069470c6574219a79aba315e26c76db49d347678a5a273f1c41c9a
$ pip3 install onnxruntime_gpu-1.20.2-cp310-cp310-linux_aarch64.whl 
$ python3
Python 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import onnxruntime as ort
>>> ort.get_available_providers()
['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']

Thanks.

[tcemadhavan]

Yes, It is working. Thanks!

[indiabrainchip]

May I know, how you confirmed that it is running on GPU?

import onnx
import onnx.helper as helper
import onnxruntime as ort
import numpy as np
import time
import os

def create_matmul_onnx(file_name):
    X = helper.make_tensor_value_info("X", onnx.TensorProto.FLOAT, [1024, 1024])
    W = helper.make_tensor_value_info("W", onnx.TensorProto.FLOAT, [1024, 1024])
    Y = helper.make_tensor_value_info("Y", onnx.TensorProto.FLOAT, [1024, 1024])

    node = helper.make_node("MatMul", ["X", "W"], ["Y"])
    graph = helper.make_graph([node], "MatMulGraph", [X, W], [Y])

    opset_import = [helper.make_operatorsetid("", 20)]
    model = helper.make_model(graph, producer_name="matmul_test", opset_imports=opset_import)
    model.ir_version = 9

    onnx.save(model, file_name)
    print(f"ONNX model saved as {file_name} with IR version 9 and Opset 20.")

def run_matmul(model_path, provider_name):
    try:
        # Suppress ONNX Runtime Logs
        os.environ["ORT_LOG_SEVERITY"] = "0" # 0 = Verbose, 1 = Info, 2 = Warning, 3 = Error, 4 = Fatal
        session_options = ort.SessionOptions()
        session_options.log_severity_level = 2
        session_options.log_verbosity_level = 2

        ort_session = ort.InferenceSession(model_path, session_options, providers=[provider_name])
        print("Session Providers:", ort_session.get_providers())
        
        print(f"🚀 Using {provider_name} for inference.")

        X_data = np.random.rand(1024, 1024).astype(np.float32)
        W_data = np.random.rand(1024, 1024).astype(np.float32)

        print("Running MatMul...")
        start_time = time.time()
        outputs = ort_session.run(None, {"X": X_data, "W": W_data})
        end_time = time.time()

        print(f"Output Shape: {outputs[0].shape}")
        print(f"Execution Time on {provider_name}: {end_time - start_time:.6f} seconds\n")

    except Exception as e:
        print(f"Error during inference on {provider_name}:", e)

if __name__ == "__main__":
    create_matmul_onnx("matmul.onnx")

    providers = ort.get_available_providers()
    print("Available Providers:", providers)

    if "CUDAExecutionProvider" in providers:
        run_matmul("matmul.onnx", "CUDAExecutionProvider")
    else:
        print("❗ CUDAExecutionProvider not available. Check your GPU and CUDA setup.")

    if "CPUExecutionProvider" in providers:
        run_matmul("matmul.onnx", "CPUExecutionProvider")
    else:
        print("❗ CPUExecutionProvider not available. Check your ONNX Runtime installation.")

I tried the code. The first model ran on CUDA, and the second on the CPU. However, I see that the CPU is faster compared to the GPU.

could you please check and confirm?

Also, are you seeing any warnings similar to below?
/usr/lib/aarch64-linux-gnu/libcudnn.so.8: version `libcudnn.so.8’ not found (required by /home/jetson/.conda/envs/test/lib/python3.10/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so)

[AastaLLL]

Hi, @indiabrainchip

Since the environment (JetPack 5 vs JetPack 6) and issue (performance vs functionality) are different.
Could you file a new topic for your question?

Thanks.

[indiabrainchip]

Later with the below package installation commands, I’m now able to run the basic functionality test (new topic not reqd).

wget https://pypi.jetson-ai-lab.dev/jp6/cu126/+f/f6e/2baa664069470/onnxruntime_gpu-1.20.2-cp310-cp310-linux_aarch64.whl#sha256=f6e2baa664069470c6574219a79aba315e26c76db49d347678a5a273f1c41c9a
pip install onnxruntime_gpu-1.20.2-cp310-cp310-linux_aarch64.whl

Thanks.