Hi all, it is regarding getting to know practical implementation of cuda graph in tensorrt.
Though the sample program code for trtexec is present in the directory
/usr/src/tensorrt/samples/common/
It is complex and has become difficult understand for writing a separate code.
I request you to please provide one simple program both in python and c++ that makes use of CUDA Graph.
Using this concept I want to make modifications to the sample.py files present in the directory
/usr/src/tensorrt/samples/python/int8_caffe_mnist/
/usr/src/tensorrt/samples/python/network_api_pytorch_mnist/
Hi SivaRamaKrishnan, I have read it but with this we cannot get practical information to implement. I need a simple python sample for inference using cuda graph. That will solve my problem. Otherwise let me know github link to any tensorrt samples making use of cuda graph.
Dear @trivedi.nagaraj,
I don’t see any sample using cuda graph except trtexec.
Did you check adding cudaStreamBeginCapture , cudaStreamEndCapture, cudaGraphInstantiate, cudaGraphLaunch functions to your code as given in Developer Guide :: NVIDIA Deep Learning TensorRT Documentation . Please see if you notice any CUDA error returned for any of these functions?
Dear @trivedi.nagaraj,
I see below code snippet for python in the link
Python
from cuda import cudart
err, stream = cudart.cudaStreamCreate()
# Call execute_async_v3() once after an input shape change to update internal state.
context.execute_async_v3(stream);
# Capture a CUDA graph instance
cudaStreamBeginCapture(stream, cudart.cudaStreamCaptureModeGlobal)
context.execute_async_v3(stream)
err, graph = cudart.cudaStreamEndCapture(stream)
err, instance = cudart.cudaGraphInstantiate(graph, 0)
# To run inferences, launch the graph instead of calling execute_async_v3().
for i in range(iterations):
cudart.cudaGraphLaunch(instance, stream)
cudart.cudaStreamSynchronize(stream)
Hi ShivaRamaKrishna, I tried executing it but it thrown the error as
from cuda import cudart
ModuleNotFoundError: No module named ‘cuda’
When I tried echoing the contents LD_LIBRARY_PATH it showed the following output
$ echo $LD_LIBRARY_PATH
/usr/lib/llvm-8/lib:/usr/lib/llvm-8/lib:/usr/local/cuda-10.2/lib64{LD_LIBRARY_PATH:+:}
Hi ShivaramaKrishnan, it is already installed long back and when tried installing it said
Requirement already satisfied. Now let me know even though it is already installed much before what could be the actual problem for ModuleNotFoundError: No module named ‘cuda’ (which I was posted before)
Below is its output of the pycuda installation from the link you have provided.
mistral@ubuntu:~$ sudo apt-get install python3-pip
[sudo] password for mistral:
Reading package lists… Done
Building dependency tree
Reading state information… Done
python3-pip is already the newest version (9.0.1-2.3~ubuntu1.18.04.8).
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
mistral@ubuntu:~$ pip3 install Cython
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: Cython in ./.local/lib/python3.6/site-packages (3 .0.4)
mistral@ubuntu:~$ pip3 install pycuda --user
Requirement already satisfied: pycuda in ./.local/lib/python3.6/site-packages (2 020.1)
Requirement already satisfied: mako in ./.local/lib/python3.6/site-packages (fro m pycuda) (1.1.6)
Requirement already satisfied: decorator>=3.2.0 in /usr/lib/python3/dist-package s (from pycuda) (4.1.2)
Requirement already satisfied: pytools>=2011.2 in ./.local/lib/python3.6/site-pa ckages (from pycuda) (2022.1.12)
Requirement already satisfied: appdirs>=1.4.0 in ./.local/lib/python3.6/site-pac kages (from pycuda) (1.4.4)
Requirement already satisfied: dataclasses>=0.7 in ./.local/lib/python3.6/site-p ackages (from pytools>=2011.2->pycuda) (0.8)
Requirement already satisfied: platformdirs>=2.2.0 in ./.local/lib/python3.6/sit e-packages (from pytools>=2011.2->pycuda) (2.4.0)
Requirement already satisfied: typing-extensions>=4.0 in ./.local/lib/python3.6/ site-packages (from pytools>=2011.2->pycuda) (4.1.1)
Requirement already satisfied: MarkupSafe>=0.9.2 in ./.local/lib/python3.6/site- packages (from mako->pycuda) (1.0)
mistral@ubuntu:~$
Hi SivaramaKrishna, thank you for the information you have provided. I tried it but could not succeed in installing. Here the error message I have got
pip3 install cuda-python
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement cuda-python (from versions: none)
ERROR: No matching distribution found for cuda-python