Description
I am trying to run YoloV4 on Orin NX using the steps that I followed for other Jetson boards, e.g, Nano, Tx2 with Jetpack 4.x revisions. The release that I am using for Orin NX is 5.1.2 which comes with TensorRT 8.5.2.2, CuDNN 8.6 and CUDA 11.4. I prepapred both Yolo and Tiny Yolo ONNX with NMS on a host machine running Ubuntu 20.04 with exact same versions of TensorRT, CuDNN and CUDA and then use them to build TRT engine on Orin NX. I am using OpenCV version 4.4.0.
The issue that I am facing is with inferencing. I converted both the ONNX file to trt engine. However, when I ran “$ …/bin/yolov4 --fp16” (yolo_deepstream/tensorrt_yolov4 at main · NVIDIA-AI-IOT/yolo_deepstream · GitHub) to convert ONNX to engine and test inference, Tiny YoloV4 is throwing the following exception (see the error at the end of the output log). However, the full YoloV4 is doing just fine. Could anyone help me finding what the issue is?
Thanks in advance.
$ …/bin/yolov4 --fp16
&&&& RUNNING TensorRT.sample_yolo [TensorRT v8502] # …/bin/yolov4 --fp16
There are 0 coco images to process
[01/07/2024-17:23:37] [I] Building and running a GPU inference engine for Yolo
[01/07/2024-17:23:37] [I] [TRT] [MemUsageChange] Init CUDA: CPU +221, GPU +0, now: CPU 250, GPU 5648 (MiB)
[01/07/2024-17:23:40] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +303, GPU +430, now: CPU 575, GPU 6100 (MiB)
[01/07/2024-17:23:40] [I] Parsing ONNX file: …/data/yolov4.onnx
[01/07/2024-17:23:40] [I] [TRT] ----------------------------------------------------------------
[01/07/2024-17:23:40] [I] [TRT] Input filename: …/data/yolov4.onnx
[01/07/2024-17:23:40] [I] [TRT] ONNX IR version: 0.0.4
[01/07/2024-17:23:40] [I] [TRT] Opset version: 11
[01/07/2024-17:23:40] [I] [TRT] Producer name:
[01/07/2024-17:23:40] [I] [TRT] Producer version:
[01/07/2024-17:23:40] [I] [TRT] Domain:
[01/07/2024-17:23:40] [I] [TRT] Model version: 0
[01/07/2024-17:23:40] [I] [TRT] Doc string:
[01/07/2024-17:23:40] [I] [TRT] ----------------------------------------------------------------
[01/07/2024-17:23:40] [W] [TRT] onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[01/07/2024-17:23:40] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[01/07/2024-17:23:40] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[01/07/2024-17:23:40] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[01/07/2024-17:23:40] [W] [TRT] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[01/07/2024-17:23:41] [I] [TRT] No importer registered for op: BatchedNMS_TRT. Attempting to import as plugin.
[01/07/2024-17:23:41] [I] [TRT] Searching for plugin: BatchedNMS_TRT, plugin_version: 1, plugin_namespace:
[01/07/2024-17:23:41] [W] [TRT] builtin_op_importers.cpp:5243: Attribute scoreBits not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[01/07/2024-17:23:41] [W] [TRT] builtin_op_importers.cpp:5243: Attribute caffeSemantics not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[01/07/2024-17:23:41] [I] [TRT] Successfully created plugin: BatchedNMS_TRT
[01/07/2024-17:23:41] [I] Building TensorRT engine…/data/yolov4.engine
[01/07/2024-17:23:41] [I] [TRT] ---------- Layers Running on DLA ----------
[01/07/2024-17:23:41] [I] [TRT] ---------- Layers Running on GPU ----------
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONSTANT: 886[Constant]
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONSTANT: 886[Constant]_clone_1
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONSTANT: 447[Constant]
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONSTANT: 447[Constant]_clone_1
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_119 + node_of_120
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(node_of_121)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_122 + node_of_123 + PWN(node_of_124)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_125 + node_of_126 + PWN(node_of_127)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_143 + node_of_144 + PWN(node_of_145)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_146 + node_of_147 + PWN(node_of_148)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 145 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_150 + node_of_151 + PWN(node_of_152)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 127 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POOLING: node_of_154
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_155 + node_of_156 + PWN(node_of_157)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_173 + node_of_174 + PWN(node_of_175)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_176 + node_of_177 + PWN(node_of_178)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 175 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_180 + node_of_181 + PWN(node_of_182)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 157 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POOLING: node_of_184
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_185 + node_of_186 + PWN(node_of_187)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_203 + node_of_204 + PWN(node_of_205)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_206 + node_of_207 + PWN(node_of_208)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 205 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_210 + node_of_211 + PWN(node_of_212)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 187 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 212 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POOLING: node_of_214
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_215 + node_of_216 + PWN(node_of_217)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_218 + node_of_219 + PWN(node_of_220)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_221 + node_of_222 + PWN(node_of_223)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_590 + node_of_591 + PWN(node_of_592)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_224
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 229 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 249 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 269 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 234 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 254 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 274 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 239 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 259 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 279 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 244 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 264 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 284 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(node_of_347)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CAST: node_of_443
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CAST: node_of_448
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: (Unnamed Layer* 485) [Shuffle]
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: (Unnamed Layer* 487) [Shuffle]
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_614
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(node_of_342), 343 + (Unnamed Layer* 379) [Shuffle] + node_of_344), PWN(345 + (Unnamed Layer* 401) [Shuffle], node_of_346))
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 369 + (Unnamed Layer* 404) [Shuffle] + node_of_370
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 376 + (Unnamed Layer* 407) [Shuffle] + node_of_377
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 397 + (Unnamed Layer* 410) [Shuffle] + node_of_398
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 404 + (Unnamed Layer* 413) [Shuffle] + node_of_405
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 425 + (Unnamed Layer* 416) [Shuffle] + node_of_426
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 432 + (Unnamed Layer* 419) [Shuffle] + node_of_433
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_322 + node_of_323
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 355 + node_of_356
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 362 + node_of_363
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 383 + node_of_384
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 390 + node_of_391
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 411 + node_of_412
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 418 + node_of_419
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_303
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SLICE: node_of_636
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(node_of_348)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_341
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_658
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_658_copy_output
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 434 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 435 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_588
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] ELEMENTWISE: node_of_444
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] ELEMENTWISE: node_of_449
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(node_of_349), node_of_589)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_660 + node_of_661 + PWN(node_of_662)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_472_copy_input
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_472
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_518_copy_input
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_518
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_495_copy_input
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_495
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_541_copy_input
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_541
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CONVOLUTION: node_of_663
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(542 + (Unnamed Layer* 542) [Shuffle], node_of_543), node_of_544)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(545 + (Unnamed Layer* 545) [Shuffle], node_of_546), node_of_547)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] ELEMENTWISE: node_of_548
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] ELEMENTWISE: node_of_549
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 668 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 688 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 708 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 673 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 693 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 713 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 678 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 698 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 718 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 683 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 703 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 723 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 544 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 547 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(node_of_786)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CAST: node_of_882
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] CAST: node_of_887
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_570
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_570_copy_output
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: (Unnamed Layer* 742) [Shuffle]
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: (Unnamed Layer* 744) [Shuffle]
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(node_of_781), 782 + (Unnamed Layer* 647) [Shuffle] + node_of_783), PWN(784 + (Unnamed Layer* 668) [Shuffle], node_of_785))
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 808 + (Unnamed Layer* 671) [Shuffle] + node_of_809
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 815 + (Unnamed Layer* 674) [Shuffle] + node_of_816
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 836 + (Unnamed Layer* 677) [Shuffle] + node_of_837
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 843 + (Unnamed Layer* 680) [Shuffle] + node_of_844
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 864 + (Unnamed Layer* 683) [Shuffle] + node_of_865
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 871 + (Unnamed Layer* 686) [Shuffle] + node_of_872
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_761 + node_of_762
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 794 + node_of_795
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 801 + node_of_802
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 822 + node_of_823
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 829 + node_of_830
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 850 + node_of_851
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SCALE: 857 + node_of_858
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_742
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(node_of_787)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_780
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 873 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 874 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_1027
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] ELEMENTWISE: node_of_883
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] ELEMENTWISE: node_of_888
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(node_of_788), node_of_1028)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_911_copy_input
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_911
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_957_copy_input
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_957
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_934_copy_input
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_934
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_980_copy_input
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_980
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(981 + (Unnamed Layer* 757) [Shuffle], node_of_982), node_of_983)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(984 + (Unnamed Layer* 760) [Shuffle], node_of_985), node_of_986)
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] ELEMENTWISE: node_of_987
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] ELEMENTWISE: node_of_988
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 983 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: 986 copy
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] SHUFFLE: node_of_1009
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] COPY: node_of_1009_copy_output
[01/07/2024-17:23:41] [I] [TRT] [GpuLayer] PLUGIN_V2: node_of_num_detections
[01/07/2024-17:23:43] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +534, GPU +834, now: CPU 1157, GPU 6980 (MiB)
[01/07/2024-17:23:43] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +83, GPU +124, now: CPU 1240, GPU 7104 (MiB)
[01/07/2024-17:23:43] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[01/07/2024-17:26:09] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size will enable more tactics, please check verbose output for requested sizes.
[01/07/2024-17:30:53] [I] [TRT] Total Activation Memory: 30029824
[01/07/2024-17:30:53] [I] [TRT] Detected 1 inputs and 4 output network tensors.
[01/07/2024-17:30:53] [I] [TRT] Total Host Persistent Memory: 73648
[01/07/2024-17:30:53] [I] [TRT] Total Device Persistent Memory: 82432
[01/07/2024-17:30:53] [I] [TRT] Total Scratch Memory: 512
[01/07/2024-17:30:53] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 35 MiB, GPU 48 MiB
[01/07/2024-17:30:53] [I] [TRT] [BlockAssignment] Started assigning block shifts. This will take 110 steps to complete.
[01/07/2024-17:30:53] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 30.8437ms to assign 15 blocks to 110 nodes requiring 7188480 bytes.
[01/07/2024-17:30:53] [I] [TRT] Total Activation Memory: 7188480
[01/07/2024-17:30:53] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +5, now: CPU 1597, GPU 7771 (MiB)
[01/07/2024-17:30:53] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +1, GPU +9, now: CPU 1598, GPU 7780 (MiB)
[01/07/2024-17:30:53] [W] [TRT] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[01/07/2024-17:30:53] [W] [TRT] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[01/07/2024-17:30:53] [W] [TRT] Check verbose logs for the list of affected weights.
[01/07/2024-17:30:53] [W] [TRT] - 19 weights are affected by this issue: Detected subnormal FP16 values.
[01/07/2024-17:30:53] [W] [TRT] - 5 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
[01/07/2024-17:30:53] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +11, GPU +16, now: CPU 11, GPU 16 (MiB)
[01/07/2024-17:30:53] [I] TRT Engine file saved to: …/data/yolov4.engine
4
[01/07/2024-17:30:54] [I] Loading or building yolo model done
[01/07/2024-17:30:54] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +7, now: CPU 1255, GPU 7628 (MiB)
[01/07/2024-17:30:54] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +1, GPU +7, now: CPU 1256, GPU 7629 (MiB)
[01/07/2024-17:30:54] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +7, now: CPU 11, GPU 23 (MiB)
[01/07/2024-17:30:54] [E] [TRT] 1: [executionContext.cpp::executeInternal::1010] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
Time consumed in preProcess: 0
Time consumed in model: 0
Time consumed in postProcess: 0
[01/07/2024-17:30:54] [E] [TRT] 1: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaStream::47] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:54] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [I] Inference of yolo model done
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[01/07/2024-17:30:55] [E] [TRT] 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
CUDA_ERROR_ILLEGAL_ADDRESS
terminate called after throwing an instance of ‘std::exception’
what(): std::exception
Aborted (core dumped)