Cuda Error depending on batchsize of engine file

Hello,

I am using the Nvidia Jetson AGX Developer Kit:

  • JetPack Version: 5.1
  • TensorRT Version: 8.5.2.2
  • Torch: 1.14
  • Torchvision: 0.14

Futhermore I am using a yolov5 small modell trained in pytorch for fp16 and imagesize (416,416). Now I wanted to convert the pt files to engine files and run inference with them. So I used the export.py script from ultralytics/yolov5 repository and convert the yolov5 modell to onnx. After that a converted the modell from onnx to engine on two different ways.

  1. I used the buildin /usr/src/tenorrt/bin/trtexec command to convert the create onnx files to engine files using the following command:

/usr/src/tensorrt/bin/trtexec --onnx=/path/to/onnx --fp16

  1. I used the run method of the utralytics/yolov5/export.py script to convert to fp16 (half) engine

I have done this for batchsize 1,2,4,8,12,16,32. Than a made an inference with an big image (4864,3232) and crop down the images to (416,416) with a little overlap and put those in batches and feed them to the neural network. For batchsize 1,2,4 it work fine. But for batchsize 8,12,16,32 Cuda throws a big error message:


[03/03/2023-12:23:19] [TRT] [E] 1: [reformat.cpp::executeCutensor::387] Error Code 1: CuTensor (Internal cuTensor permutate execute failed)
[03/03/2023-12:23:19] [TRT] [E] 1: [checkMacros.cpp::catchCudaError::202] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
Catched Exception:  CUDA error: an illegal memory access was encountered
Traceback (most recent call last):
  File "start_inference.py", line 461, in start_test_inferenz
    self.detector.run_warm_up(cycles = 2)
  File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/helper.py", line 49, in log_prt
    result = method(*args, **kw)
  File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/helper.py", line 133, in timed
    result = method(*args, **kw)
  File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/detect.py", line 304, in run_warm_up
    r = self.obj_detect_tiled(
  File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/helper.py", line 133, in timed
    result = method(*args, **kw)
  File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/detect.py", line 326, in obj_detect_tiled
    r = self.detect_tiled(self.net, self.par, img_in,  
  File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/yolov3_detect.py", line 220, in detect_tiled
    torch.cuda.synchronize() #?        
  File "/home/edpagx/Desktop/ProjectNN/jetson/Enviroments/detection_model/lib/python3.8/site-packages/torch/cuda/__init__.py", line 577, in synchronize
    return torch._C._cuda_synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "start_inference.py", line 1525, in <module>
    y.start_test_inferenz(evaluate_res=True, safe_plot_path="/media/edpagx/SHD1/Models/ships/out_images/", log_to_file=False, plot_results=True)
  File "start_inference.py", line 515, in start_test_inferenz
    self.pwhandler.stop_powermessurement(prozess=prozess, delete_file=True, delete_dir=True)
UnboundLocalError: local variable 'prozess' referenced before assignment
[03/03/2023-12:23:20] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaStream::47] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
CUDA_ERROR_ILLEGAL_ADDRESS
terminate called after throwing an instance of 'std::exception'
  what():  std::exception
Aborted (core dumped)

Here is the code, with lead to the error messages. It is only a part of the real code with does the inference for a larger scaled image.

import numpy as np
import torch

from torch.cuda.amp import autocast
import torchvision.transforms as transforms

from KI_public.NNP_public.helper import timeit, tile_coords, make_tiles, scale_batch

from KI_public.yolov5.utils.general import non_max_suppression 

# =============================================================================
# 
# =============================================================================
@timeit
def apply(model, batch, log_time = None): 
    #inference with timing
    with torch.no_grad():
        prediction = model(batch) #, targets = None
    return prediction

# =============================================================================
# 
# =============================================================================
#@timeit
def _fill_rlist(predictions, imgs, tiles):
    
    """convert all results, NMS is assumed to be done before"""
    
    rlist = []
    for i,(pred,img) in enumerate(zip(predictions,imgs)):
        r = {'img_orig': img, 
              'res_pred_boxes': [],
              'res_pred_class': [],
              'res_pred_score': [],
              'tiles': tiles, 
              }
        if pred is not None and  pred.nelement() > 0  and pred != []: #pred != [], ?
            r ['res_pred_boxes'] = pred[:,0:4].cpu().round().tolist()
            r ['res_pred_class'] = pred[:,-1].cpu().int().tolist() 
            r ['res_pred_score'] = (100*pred[:,4]).cpu().round().int().tolist()
    
        rlist.append(r)
                 
    return rlist

# =============================================================================
# 
# =============================================================================
#@timeit #don't count twice
def apply_sc_offset(all_predictions, sc, offset, log_time = None):
  
    # Convert nx4 boxes from [x, y, w, h] to sc*[x, y, w, h]+offset
    # works inplace, assume that the memory is copied before

    x = torch.cat(all_predictions)   
    nn = x.shape[1]
   
    x = x.view([-1,x.shape[-1]])
    for i in range(offset.shape[0]): #for all tiles, to do: remove the loop by further expands ...
        ib = i*nn
        ie = ib+nn       
        x[ib:ie, 0:4]  = sc.expand(nn,4) * x[ib:ie, 0:4] + offset[i].expand(nn,4)  # top left xy, wh
    
    return x.unsqueeze(0)       
 
# =============================================================================
#    
# =============================================================================

@timeit
def apply_sc_offset_nms(all_predictions, sc, offset,
                        conf_thres=0.25, iou_thres=0.45, 
                        classes=None, agnostic=False, multi_label=False,
                        labels=(), 
                        max_det=300,
                        min_wh = 2,
                        max_wh = 7680, 
                        log_time = None):
    
    
    all_predictions = apply_sc_offset(all_predictions,sc, offset) 
    
    all_predictions = non_max_suppression(all_predictions, conf_thres, iou_thres,
                                                classes = classes, 
                                                max_det = max_det,
                                                min_wh = min_wh, 
                                                max_wh = max_wh,
                                              #  log_time = log_time,
                                            )  
    
    return all_predictions

# =============================================================================
# 
# =============================================================================
def detect_tiled(model, par, img_in,                    
                tiles = [],
                segm_conv = None, #unused
                log_time = None,
                ):
      
    """Tesselate the image and apply the model to the overlapping image tiles,
        provide a result for the whole image""" 
        
    if hasattr(model, 'fp16') and model.fp16: #set in DetectMultiBackend of yolov5
        input_fp16 = True
    else:  
        input_fp16 = False
  
    confidence = 0.01*par.score_threshold 
    
    if par.max_boxes_per_image == None:
        par.max_boxes_per_image = par.max_boxes_per_tile
        
    min_wh = min(par.box_min_size) if par.box_max_size != None else 2 
    max_wh = max(par.box_max_size) if par.box_max_size != None else 7680
        
    if not isinstance(img_in, torch.Tensor):
        raise TypeError
        
    if tiles == []:  
        #the tile_size here includes a scaling factor which is applied in make tiles
        #prepare crop boxes for tiling per image, could be done for images with equal sizes  
        w_h = (img_in.shape[1], img_in.shape[2])
       
        #the tiles must match the tile_size, handle small dimensions
        if w_h[0] < par.tile_size[0] or w_h[1] < par.tile_size[1]:
            print("reshape the image, at least one dimension to small", img_in.shape)
            bottom = max(0,par.tile_size[0] - w_h[0])
            right  = max(0,par.tile_size[1] - w_h[1])
            m = torch.nn.ConstantPad2d((0,right,0, bottom),0)
            img_in =  m(img_in)
            w_h = (img_in.shape[1], img_in.shape[2])
            print("new", img_in.shape)

        tiles = tile_coords(w_h, par.tile_size, par.tess_step, par.skip_frac_tiles, par.box_min_size,
                            log_time = log_time) 
     
    if tiles is None:
        return None 
    
    offset = torch.tensor([[tile[0],tile[1],0,0] for tile in tiles])
    
    sc_inv = torch.tensor([par.tile_size[0]/par.net_dims[0], par.tile_size[1]/par.net_dims[1]])
    tsf = [1./sc_inv[0].item(), 1./sc_inv[1].item()]  #we need tuple for interpolate
    
    orig_ims, tiles_s = make_tiles(img_in, tiles,
                                 use_cuda = par.tesselate_on_gpu,  
                                 img_prescale_factor = tsf, 
                                 input_fp16 = input_fp16,
                                 log_time = log_time, as_list = False) #True)   
    if par.use_cuda:
        sc_inv = sc_inv.cuda()
        offset = offset.cuda()
              
    # converting images to batches    
    reminder = 0
    if (len(orig_ims) % par.batch_size):
        reminder = 1   
    num_batches = len(orig_ims) // par.batch_size + reminder            
    im_batches = [orig_ims[i*par.batch_size : min((i +  1)*par.batch_size,len(orig_ims))]
                    for i in range(num_batches)] 
    
    all_predictions = []     
    add_size = 0
    for batch_num, batch in enumerate(im_batches): 
          
        #add dummy images to regularize the last batch_size (for .engine), todo: do we still need it?
        if par.regular_batch_size  and len(batch) < par.batch_size:
            add_size = par.batch_size - batch.shape[0]
            add = torch.ones([add_size, batch.shape[1],batch.shape[2], batch.shape[3] ] ) 
            if par.tesselate_on_gpu and par.use_cuda:
                add = add.cuda()  
                if input_fp16:
                    add = add.half()
            batch = torch.cat((batch, add), 0)
                    
        if not par.tesselate_on_gpu and par.use_cuda:
            batch = batch.cuda()  
       
        with autocast(enabled = par.use_mixed_prec):
          #  batch = (255*batch).to(torch.uint8).permute(0,2,3,1)
            prediction = apply(model, batch,  log_time = log_time) #raw preds
          
        if add_size > 0: #remove last invalid predictions              
            prediction = prediction[:(par.batch_size - add_size),:,:]
          #  print ("shorter last batch:", batch_num, prediction.shape) 
   
        if par.use_cuda:
            torch.cuda.synchronize() #?        
      
        if not prediction.is_cuda:
            print ("prediction not cuda!")
        all_predictions.append(prediction.detach().clone() if isinstance(prediction, torch.Tensor) else np.copy(prediction))
     
    #print (len(all_predictions), all_predictions[0].shape)
    all_predictions =  apply_sc_offset_nms(all_predictions,torch.cat([sc_inv,sc_inv]), offset, confidence, par.iou_threshold,
                                                classes = list(range(0,par.num_classes,1)), #to check,different for v3 and v5?
                                                max_det = par.max_boxes_per_image,
                                                min_wh = min_wh,
                                                max_wh = max_wh,
                                                log_time = log_time )
    
    r = _fill_rlist(all_predictions, [img_in], tiles)[0] #! _fill_rlist works for lists     
    
    return r

I would really appreciating, if someone could help me or even tell me what could be the reason for this error.

Yours
Patrick

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!