Illegal memory access using cuda graph and CUPTI

AI & Data Science Deep Learning (Training & Inference) TensorRT
, ,
1

Description

I am working on a test using the sampleOnnxMNIST from the TensorRT official samples. To optimize the performance, I have enabled CUDA Graph to execute the inference and employed CUPTI to track the kernel activities from the model.

In the test, I have observed that the sequence initTrace followed by infer works correctly. However, when I try the sequence infer followed by initTrace and then another infer, I encounter an error with code 700 which means an illegal memory access was encountered.

Environment

TensorRT Version: v8502
GPU Type: NVIDIA GeForce RTX 3070
Nvidia Driver Version: 510.39.01
CUDA Version: 11.6
CUDNN Version: 8.9.6
Operating System + Version: Ubuntu 20.04.6 LTS
Python Version (if applicable): 3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

trt_samples.tar.gz (15.8 MB)

Steps To Reproduce

  • untar the sample code tar xvzf trt_sample.tar.gz
  • cd to the directory trt_samples/sampleOnnxMNIST
  • build the sample make -j8
  • run the sample ../../bin/sample_onnx_mnist
  • sample will abort and occur the error Cuda failure: an illegal memory access was encountered