Hello,
I am using the Nvidia Jetson AGX Developer Kit:
- JetPack Version: 5.1
- TensorRT Version: 8.5.2.2
- Torch: 1.14
- Torchvision: 0.14
Futhermore I am using a yolov5 small modell trained in pytorch for fp16 and imagesize (416,416). Now I wanted to convert the pt files to engine files and run inference with them. So I used the export.py script from ultralytics/yolov5 repository and convert the yolov5 modell to onnx. After that a converted the modell from onnx to engine on two different ways.
- I used the buildin /usr/src/tenorrt/bin/trtexec command to convert the create onnx files to engine files using the following command:
/usr/src/tensorrt/bin/trtexec --onnx=/path/to/onnx --fp16
- I used the run method of the utralytics/yolov5/export.py script to convert to fp16 (half) engine
I have done this for batchsize 1,2,4,8,12,16,32. Than a made an inference with an big image (4864,3232) and crop down the images to (416,416) with a little overlap and put those in batches and feed them to the neural network. For batchsize 1,2,4 it work fine. But for batchsize 8,12,16,32 Cuda throws a big error message:
[03/03/2023-12:23:19] [TRT] [E] 1: [reformat.cpp::executeCutensor::387] Error Code 1: CuTensor (Internal cuTensor permutate execute failed)
[03/03/2023-12:23:19] [TRT] [E] 1: [checkMacros.cpp::catchCudaError::202] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
Catched Exception: CUDA error: an illegal memory access was encountered
Traceback (most recent call last):
File "start_inference.py", line 461, in start_test_inferenz
self.detector.run_warm_up(cycles = 2)
File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/helper.py", line 49, in log_prt
result = method(*args, **kw)
File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/helper.py", line 133, in timed
result = method(*args, **kw)
File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/detect.py", line 304, in run_warm_up
r = self.obj_detect_tiled(
File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/helper.py", line 133, in timed
result = method(*args, **kw)
File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/detect.py", line 326, in obj_detect_tiled
r = self.detect_tiled(self.net, self.par, img_in,
File "/home/edpagx/Desktop/ProjectNN/jetson/optimization_nn/yolo_runtime/inference/../../yolo_runtime/KI_Scripts/scripts/yolov3_detect.py", line 220, in detect_tiled
torch.cuda.synchronize() #?
File "/home/edpagx/Desktop/ProjectNN/jetson/Enviroments/detection_model/lib/python3.8/site-packages/torch/cuda/__init__.py", line 577, in synchronize
return torch._C._cuda_synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "start_inference.py", line 1525, in <module>
y.start_test_inferenz(evaluate_res=True, safe_plot_path="/media/edpagx/SHD1/Models/ships/out_images/", log_to_file=False, plot_results=True)
File "start_inference.py", line 515, in start_test_inferenz
self.pwhandler.stop_powermessurement(prozess=prozess, delete_file=True, delete_dir=True)
UnboundLocalError: local variable 'prozess' referenced before assignment
[03/03/2023-12:23:20] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaStream::47] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:20] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[03/03/2023-12:23:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
CUDA_ERROR_ILLEGAL_ADDRESS
terminate called after throwing an instance of 'std::exception'
what(): std::exception
Aborted (core dumped)
Here is the code, with lead to the error messages. It is only a part of the real code with does the inference for a larger scaled image.
import numpy as np
import torch
from torch.cuda.amp import autocast
import torchvision.transforms as transforms
from KI_public.NNP_public.helper import timeit, tile_coords, make_tiles, scale_batch
from KI_public.yolov5.utils.general import non_max_suppression
# =============================================================================
#
# =============================================================================
@timeit
def apply(model, batch, log_time = None):
#inference with timing
with torch.no_grad():
prediction = model(batch) #, targets = None
return prediction
# =============================================================================
#
# =============================================================================
#@timeit
def _fill_rlist(predictions, imgs, tiles):
"""convert all results, NMS is assumed to be done before"""
rlist = []
for i,(pred,img) in enumerate(zip(predictions,imgs)):
r = {'img_orig': img,
'res_pred_boxes': [],
'res_pred_class': [],
'res_pred_score': [],
'tiles': tiles,
}
if pred is not None and pred.nelement() > 0 and pred != []: #pred != [], ?
r ['res_pred_boxes'] = pred[:,0:4].cpu().round().tolist()
r ['res_pred_class'] = pred[:,-1].cpu().int().tolist()
r ['res_pred_score'] = (100*pred[:,4]).cpu().round().int().tolist()
rlist.append(r)
return rlist
# =============================================================================
#
# =============================================================================
#@timeit #don't count twice
def apply_sc_offset(all_predictions, sc, offset, log_time = None):
# Convert nx4 boxes from [x, y, w, h] to sc*[x, y, w, h]+offset
# works inplace, assume that the memory is copied before
x = torch.cat(all_predictions)
nn = x.shape[1]
x = x.view([-1,x.shape[-1]])
for i in range(offset.shape[0]): #for all tiles, to do: remove the loop by further expands ...
ib = i*nn
ie = ib+nn
x[ib:ie, 0:4] = sc.expand(nn,4) * x[ib:ie, 0:4] + offset[i].expand(nn,4) # top left xy, wh
return x.unsqueeze(0)
# =============================================================================
#
# =============================================================================
@timeit
def apply_sc_offset_nms(all_predictions, sc, offset,
conf_thres=0.25, iou_thres=0.45,
classes=None, agnostic=False, multi_label=False,
labels=(),
max_det=300,
min_wh = 2,
max_wh = 7680,
log_time = None):
all_predictions = apply_sc_offset(all_predictions,sc, offset)
all_predictions = non_max_suppression(all_predictions, conf_thres, iou_thres,
classes = classes,
max_det = max_det,
min_wh = min_wh,
max_wh = max_wh,
# log_time = log_time,
)
return all_predictions
# =============================================================================
#
# =============================================================================
def detect_tiled(model, par, img_in,
tiles = [],
segm_conv = None, #unused
log_time = None,
):
"""Tesselate the image and apply the model to the overlapping image tiles,
provide a result for the whole image"""
if hasattr(model, 'fp16') and model.fp16: #set in DetectMultiBackend of yolov5
input_fp16 = True
else:
input_fp16 = False
confidence = 0.01*par.score_threshold
if par.max_boxes_per_image == None:
par.max_boxes_per_image = par.max_boxes_per_tile
min_wh = min(par.box_min_size) if par.box_max_size != None else 2
max_wh = max(par.box_max_size) if par.box_max_size != None else 7680
if not isinstance(img_in, torch.Tensor):
raise TypeError
if tiles == []:
#the tile_size here includes a scaling factor which is applied in make tiles
#prepare crop boxes for tiling per image, could be done for images with equal sizes
w_h = (img_in.shape[1], img_in.shape[2])
#the tiles must match the tile_size, handle small dimensions
if w_h[0] < par.tile_size[0] or w_h[1] < par.tile_size[1]:
print("reshape the image, at least one dimension to small", img_in.shape)
bottom = max(0,par.tile_size[0] - w_h[0])
right = max(0,par.tile_size[1] - w_h[1])
m = torch.nn.ConstantPad2d((0,right,0, bottom),0)
img_in = m(img_in)
w_h = (img_in.shape[1], img_in.shape[2])
print("new", img_in.shape)
tiles = tile_coords(w_h, par.tile_size, par.tess_step, par.skip_frac_tiles, par.box_min_size,
log_time = log_time)
if tiles is None:
return None
offset = torch.tensor([[tile[0],tile[1],0,0] for tile in tiles])
sc_inv = torch.tensor([par.tile_size[0]/par.net_dims[0], par.tile_size[1]/par.net_dims[1]])
tsf = [1./sc_inv[0].item(), 1./sc_inv[1].item()] #we need tuple for interpolate
orig_ims, tiles_s = make_tiles(img_in, tiles,
use_cuda = par.tesselate_on_gpu,
img_prescale_factor = tsf,
input_fp16 = input_fp16,
log_time = log_time, as_list = False) #True)
if par.use_cuda:
sc_inv = sc_inv.cuda()
offset = offset.cuda()
# converting images to batches
reminder = 0
if (len(orig_ims) % par.batch_size):
reminder = 1
num_batches = len(orig_ims) // par.batch_size + reminder
im_batches = [orig_ims[i*par.batch_size : min((i + 1)*par.batch_size,len(orig_ims))]
for i in range(num_batches)]
all_predictions = []
add_size = 0
for batch_num, batch in enumerate(im_batches):
#add dummy images to regularize the last batch_size (for .engine), todo: do we still need it?
if par.regular_batch_size and len(batch) < par.batch_size:
add_size = par.batch_size - batch.shape[0]
add = torch.ones([add_size, batch.shape[1],batch.shape[2], batch.shape[3] ] )
if par.tesselate_on_gpu and par.use_cuda:
add = add.cuda()
if input_fp16:
add = add.half()
batch = torch.cat((batch, add), 0)
if not par.tesselate_on_gpu and par.use_cuda:
batch = batch.cuda()
with autocast(enabled = par.use_mixed_prec):
# batch = (255*batch).to(torch.uint8).permute(0,2,3,1)
prediction = apply(model, batch, log_time = log_time) #raw preds
if add_size > 0: #remove last invalid predictions
prediction = prediction[:(par.batch_size - add_size),:,:]
# print ("shorter last batch:", batch_num, prediction.shape)
if par.use_cuda:
torch.cuda.synchronize() #?
if not prediction.is_cuda:
print ("prediction not cuda!")
all_predictions.append(prediction.detach().clone() if isinstance(prediction, torch.Tensor) else np.copy(prediction))
#print (len(all_predictions), all_predictions[0].shape)
all_predictions = apply_sc_offset_nms(all_predictions,torch.cat([sc_inv,sc_inv]), offset, confidence, par.iou_threshold,
classes = list(range(0,par.num_classes,1)), #to check,different for v3 and v5?
max_det = par.max_boxes_per_image,
min_wh = min_wh,
max_wh = max_wh,
log_time = log_time )
r = _fill_rlist(all_predictions, [img_in], tiles)[0] #! _fill_rlist works for lists
return r
I would really appreciating, if someone could help me or even tell me what could be the reason for this error.
Yours
Patrick