./rtSafe/safeContext.cpp (133) - Cudnn Error in configure: 7 (CUDNN_STATUS_MAPPING_ERROR)

Hi,

I have converted my Pytorch model to ONNX and to TRT(with Custom Plugins).
But I am using another TRT(2) Model whose output will be input to the above TRT(1) Model. I am serving those two TRT models using Triton Server on Jetson Nano, I am sending the request from my Laptop to Jetson Nano, but during the response, I am getting error in the Jetson nano terminal as:

E0311 13:59:57.029723 29688 logging.cc:43] …/rtSafe/safeContext.cpp (133) - Cudnn Error in configure: 7 (CUDNN_STATUS_MAPPING_ERROR)
E0311 13:59:57.030261 29688 logging.cc:43] FAILED_EXECUTION: std::exception

Not sure what is going wrong. what is causing an issue here?
Can you please assist me in resolving this error?

Command I used in Jetson Nano to start the serving of two models:

LD_PRELOAD=“Einsum_op.so RoI_Align.so libyolo_layer.so” ./Downloads/bin/tritonserver --model-repository=./models --min-supported-compute-capability=5.3 --log-verbose=1

Model Info:

  1. Yolov3-spp-ultralytics.
  2. SlowFast.(with two Custom Plugins).

The bounding boxes outputed by the 1st model will go as the input to the 2nd model(Slowfast).

Looking forward to the reply

Thanks,
Darshan

Hi,

Do you know the error occurs on 1st model or 2nd model?
More, could you try if the same error occurs when running TensorRT outside of the Triton Server?

Thanks.

Hi @AastaLLL,

Thanks for your reply.

I tried running Outside the Triton Server with TensorRT, it is working fine.
And also, for the first frame of the video, it is giving an output. But for the next frame, it is a throwing error. Not sure what is going wrong.

Model Info:

  1. Yolov3-spp-ultralytics.
  2. SlowFast.(with two Custom Plugins).

The bounding boxes outputed by 1st model will go as the input to the 2nd model(Slowfast).

As I said, for the first frame of the video, it is giving the output. But for the next frame, it is a throwing an error.

I am attaching the Plugin Code that i have Implemented below:
Einsum Plugin Code: Einsum - Google Drive
RoI Align Plugin Code: RoI_Align - Google Drive

Thanks,
Darshan

Thanks for the sharing.

We are going to reproduce this issue internally.
Will share more information with you later.

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Hi,

To help us reproduce this, could you share the ONNX models and the detailed steps with us?
Thanks.