illegal memory exception on T4

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I am porting a C++ program from the P4 platform to the T4 platform, which is work fine on P4. But on T4 could not, and the error messages are related to illegal memory. Dos T4 not support running multiple algorithm models on a single GPU card?

P4 env info: cuda9 + cudnn7 + tensorRT3
T4 env info: cuda10.0 + cudnn7.3 + tensorRT5

if anybody has some idea, please let me know.Thanks !!!

2

Hi,

Can you please provide a small repro package with code and other necessary data so I can further debug this? It is possible that code from TensorRT 3 has since been deprecated in TensorRT 5.

Thanks,
NVIDIA Enterprise Support

3

I have some models in Caffe and then optimized by TensorRT. They are running well in the same thread but I am having some problems when using mulit-threads，each Thread run a model. I am pasting the error just below:

ERROR: CUDA cask failure at execution for trt_maxwell_scudnn_128x32_relu_small_nn_v1.
ERROR: cuda/caskConvolutionLayer.cpp (256) - Cuda Error in execute: 77
ERROR: cuda/caskConvolutionLayer.cpp (256) - Cuda Error in execute: 77

I’ve generated the PLAN file from a turing uarchitecture (GeForce 1050 Ti and T4) so it sounds weird for me that part that sais “failure at execution for trt_maxwell_scudnn_128x32_relu_small_nn_v1”. Does it have any sense?

4

Hi,

Can you share a repro package with the

1. original model
2. converted model
3. scripts that you ran to convert the model
4. scripts for running the model with multiple threads
5. dataset used for models/scripts if any

so I can reproduce and debug this issue? You can private message this information to me if you don’t want to share it publicly.

Thanks,
NVIDIA Enterprise Support

5

Hi NVES_R,

Thanks for all the answers!

I’ve fixed this problem, a GpuMat copy problem.The second thread uses some GpuMat data, which is just a part of GpuMat data in the first thread.

Thanks again for your help!

6

No problem, glad you figured it out.

Thanks,
NVIDIA Enterprise Support