Please provide a sample file making use of cuda graph

Hi all, it is regarding getting to know practical implementation of cuda graph in tensorrt.

Though the sample program code for trtexec is present in the directory
/usr/src/tensorrt/samples/common/

It is complex and has become difficult understand for writing a separate code.

I request you to please provide one simple program both in python and c++ that makes use of CUDA Graph.

Using this concept I want to make modifications to the sample.py files present in the directory
/usr/src/tensorrt/samples/python/int8_caffe_mnist/
/usr/src/tensorrt/samples/python/network_api_pytorch_mnist/

Thanks and Regards

Nagaraj Trivedi

Hi all, please update me on this query.

Thanks and Regards

Nagaraj Trivedi

Dear @trivedi.nagaraj,
Did you check Developer Guide :: NVIDIA Deep Learning TensorRT Documentation?

Hi SivaRamaKrishnan, I have read it but with this we cannot get practical information to implement. I need a simple python sample for inference using cuda graph. That will solve my problem. Otherwise let me know github link to any tensorrt samples making use of cuda graph.

Thanks and Regards

Nagaraj Trivedi

Dear @trivedi.nagaraj,
I don’t see any sample using cuda graph except trtexec.

Did you check adding cudaStreamBeginCapture , cudaStreamEndCapture, cudaGraphInstantiate, cudaGraphLaunch functions to your code as given in Developer Guide :: NVIDIA Deep Learning TensorRT Documentation . Please see if you notice any CUDA error returned for any of these functions?

Hi ShivaRamKrishnan, thank you for your response. These APIs you have provided is for c++. But I have asked for python.

So far I have experimented code in the directory /usr/src/tensorrt/samples/python/network_api_pytorch_mnist/

with FP32, FP16 and INT8. Now I want to experiment it with CUDA graph. So request you to provide the sample code for performing CUDA graph in python.

Thanks and Regards

Nagaraj Trivedi

Dear @trivedi.nagaraj,
I see below code snippet for python in the link

Python
from cuda import cudart
err, stream = cudart.cudaStreamCreate()

# Call execute_async_v3() once after an input shape change to update internal state.
context.execute_async_v3(stream);

# Capture a CUDA graph instance
cudaStreamBeginCapture(stream, cudart.cudaStreamCaptureModeGlobal)
context.execute_async_v3(stream)
err, graph = cudart.cudaStreamEndCapture(stream)
err, instance = cudart.cudaGraphInstantiate(graph, 0)

# To run inferences, launch the graph instead of calling execute_async_v3().
for i in range(iterations):
    cudart.cudaGraphLaunch(instance, stream)
    cudart.cudaStreamSynchronize(stream)

Thank you ShivaRamaKrishna, I will try it and let you know the further development.

Thanks and Regards

Nagaraj Trivedi

Hi ShivaRamaKrishna, I tried executing it but it thrown the error as
from cuda import cudart
ModuleNotFoundError: No module named ‘cuda’

When I tried echoing the contents LD_LIBRARY_PATH it showed the following output
$ echo $LD_LIBRARY_PATH
/usr/lib/llvm-8/lib:/usr/lib/llvm-8/lib:/usr/local/cuda-10.2/lib64{LD_LIBRARY_PATH:+:}

Torch version is 1.11.0a0+17540c5

Thanks and Regards

Nagaraj Trivedi

Hi ShivaRamaKrishna, please update me on this. I need it to make it work quickly as it is holding my work.

Thanks and Regards

Nagaraj Trivedi

Dear @trivedi.nagaraj ,
is cuda python package installed? can you try suggestion at Install PyCuda on Jetson NX - #3 by AastaLLL

Hi ShivaramaKrishnan, it is already installed long back and when tried installing it said
Requirement already satisfied. Now let me know even though it is already installed much before what could be the actual problem for ModuleNotFoundError: No module named ‘cuda’ (which I was posted before)

Below is its output of the pycuda installation from the link you have provided.

mistral@ubuntu:~$ sudo apt-get install python3-pip
[sudo] password for mistral:
Reading package lists… Done
Building dependency tree
Reading state information… Done
python3-pip is already the newest version (9.0.1-2.3~ubuntu1.18.04.8).
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
mistral@ubuntu:~$ pip3 install Cython
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: Cython in ./.local/lib/python3.6/site-packages (3 .0.4)
mistral@ubuntu:~$ pip3 install pycuda --user
Requirement already satisfied: pycuda in ./.local/lib/python3.6/site-packages (2 020.1)
Requirement already satisfied: mako in ./.local/lib/python3.6/site-packages (fro m pycuda) (1.1.6)
Requirement already satisfied: decorator>=3.2.0 in /usr/lib/python3/dist-package s (from pycuda) (4.1.2)
Requirement already satisfied: pytools>=2011.2 in ./.local/lib/python3.6/site-pa ckages (from pycuda) (2022.1.12)
Requirement already satisfied: appdirs>=1.4.0 in ./.local/lib/python3.6/site-pac kages (from pycuda) (1.4.4)
Requirement already satisfied: dataclasses>=0.7 in ./.local/lib/python3.6/site-p ackages (from pytools>=2011.2->pycuda) (0.8)
Requirement already satisfied: platformdirs>=2.2.0 in ./.local/lib/python3.6/sit e-packages (from pytools>=2011.2->pycuda) (2.4.0)
Requirement already satisfied: typing-extensions>=4.0 in ./.local/lib/python3.6/ site-packages (from pytools>=2011.2->pycuda) (4.1.1)
Requirement already satisfied: MarkupSafe>=0.9.2 in ./.local/lib/python3.6/site- packages (from mako->pycuda) (1.0)
mistral@ubuntu:~$

Hi ShivaramaKrishna, please provide update on this.

Thanks and Regards

Nagaraj Trivedi

Dear @trivedi.nagaraj,
check if it helps to fix.

nvidia@tegra-ubuntu:~$ pip3 install cuda-python
Collecting cuda-python
  Downloading cuda_python-12.3.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (22.9 MB)
     |████████████████████████████████| 22.9 MB 40.7 MB/s
Installing collected packages: cuda-python
Successfully installed cuda-python-12.3.0
nvidia@tegra-ubuntu:~$ python3
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cuda import cudart
>>>

Hi SivaramaKrishna, thank you for the information you have provided. I tried it but could not succeed in installing. Here the error message I have got
pip3 install cuda-python
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement cuda-python (from versions: none)
ERROR: No matching distribution found for cuda-python

Thanks and Regards

Nagaraj Trivedi

Dear @trivedi.nagaraj,
Just confirming if you are using Xavier board? What is the jetpack version?

Hi SivaRamakrishnan, yes I am using Xavier board and the jetpack version is
Package: nvidia-jetpack
Version: 4.6-b199

Thanks and Regards

Nagaraj Trivedi

Dear @trivedi.nagaraj,
I could install using jetpack 5 release. Can you upgrade to latest release?

Hi Siavaramakrishna, let me try it.

Thanks and Regards

Nagaraj Trivedi

Hi SivaRamaKirshna, please provide an official link to download and install JetPack 5.

Thanks and Regards

Nagaraj Trivedi