GPU Illegal Memory Access when using pycuda/tensorrt with FP16 input

haritsah · November 12, 2022, 1:00am

Description

There was an error about illegal memory access when I tried to run the tensorrt with FP16 layer input using python script with tensorrt and pycuda similar to tensorrt_samples/yolov3_onnx. I also tried to replace pycuda.gpuarray with pcyuda.driver.mem_alloc but the same error still showing

I was able to run the quantized YOLOv6 model in Triton Inference Server using this command to build the model:

trtexec --workspace=1024 --percentile=99 --streams=1 --int8 --fp16 --avgRuns=10 --onnx=best_ckpt_bs1_graph_opt.onnx --inputIOFormats=fp16:chw

I was able to run the simple copy to device memory from terminal using pycuda.gpuarray

import numpy as np
import pycuda
import pycuda.gpuarray
pycuda.driver.init()
cuda_context = pycuda.tools.make_default_context()
stream = pycuda.driver.Stream()

data = np.random.rand(1,2,3,4).astype(np.float16)
mem = pycuda.gpuarray.empty([1,2,3,4], dtype=np.float16)

mem.set(data)

mem.set_async(data, stream)
stream.synchronize()

Error codes

[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1094, GPU 6064 (MiB)
[TensorRT] VERBOSE: Total per-runner device memory is 18651648
[TensorRT] VERBOSE: Total per-runner host memory is 146608
[TensorRT] VERBOSE: Allocated activation device memory of size 8292864
[TensorRT] INFO: [MemUsageSnapshot] ExecutionContext creation end: CPU 1095 MiB, GPU 6090 MiB
Created bindings for 0: (1, 3, 640, 640) - <class 'numpy.float16'>
Created bindings for 1: (1, 1) - <class 'numpy.int32'>
Created bindings for 2: (1, 100, 4) - <class 'numpy.float32'>
Created bindings for 3: (1, 100) - <class 'numpy.float32'>
Created bindings for 4: (1, 100) - <class 'numpy.int32'>
Initialize device buffer: (1, 3, 640, 640) - <class 'numpy.float16'>
Initialize device buffer: (1, 1) - <class 'numpy.int32'>
Initialize device buffer: (1, 100, 4) - <class 'numpy.float32'>
Initialize device buffer: (1, 100) - <class 'numpy.float32'>
Initialize device buffer: (1, 100) - <class 'numpy.int32'>
Initialize host buffer: (1, 3, 640, 640) - <class 'numpy.float16'>
Initialize host buffer: (1, 1) - <class 'numpy.int32'>
Initialize host buffer: (1, 100, 4) - <class 'numpy.float32'>
Initialize host buffer: (1, 100) - <class 'numpy.float32'>
Initialize host buffer: (1, 100) - <class 'numpy.int32'>
{'images': {'byte_size': 2457600, 'dtype': <class 'numpy.float16'>, 'shape': (1, 3, 640, 640)}}
images: (1, 3, 640, 640) - float16 - 2457600
Data: (1, 3, 640, 640) - float16 - 2457600
binding array: (1, 3, 640, 640) - float16 - 2457600
[TensorRT] ERROR: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed.)
host array: (1, 1) - int32 - 4
FAILED

===================================================================== FAILURES =====================================================================
============================================================= short test summary info ==============================================================
FAILED tests/test_trt_inference.py::test_inference - pycuda._driver.LogicError: cuMemcpyDtoHAsync failed: an illegal memory access was encountered
================================================================ 1 failed in 5.29s =================================================================
[TensorRT] ERROR: 1: [hardwareContext.cpp::terminateCommonContext::141] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [defaultAllocator.cpp::free::85] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] WARNING: Unable to determine GPU memory usage
[TensorRT] WARNING: Unable to determine GPU memory usage
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1026, GPU 0 (MiB)
[TensorRT] INTERNAL ERROR: [defaultAllocator.cpp::free::85] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaStream::455] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [resources.cpp::~ScopedCudaEvent::438] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[TensorRT] INTERNAL ERROR: [defaultAllocator.cpp::free::85] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
CUDA_ERROR_ILLEGAL_ADDRESS
terminate called after throwing an instance of 'std::runtime_error'
  what():  CUDA_ERROR_ILLEGAL_ADDRESS
Fatal Python error: Aborted

Current thread 0x00007f565cfd7740 (most recent call first):
<no Python frame>
Aborted (core dumped)

Environment

Docker: nvcr.io/nvidia/pytorch:21.10-py3, nvcr.io/nvidia/pytorch:22.06-py3
TensorRT Version: 8.0.3.4, 8.2.5
NVIDIA GPU: T4
NVIDIA Driver Version: 515.65.01
CUDA Version: 11.7
Operating System: Ubuntu

Relevant Files

I can’t share the full codes, but some relevant functions are similar to here:

Bindings - Using pagelocked_empty and pycuda.gpuarray
Engine Builder → tensorrt_samples/yolov3_onnx
But I modify the network input dtype and formats to FP16.

for idx in range(network.num_inputs):
    network.get_input(idx).allowed_formats = 1 << int(trt.TensorFormat.CHW16)
    network.get_input(idx).dtype = trt.DataType.HALF

NVES · November 15, 2022, 1:07pm

Hi,
We recommend you to raise this query in TRITON Inference Server Github instance issues section.

Thanks!

haritsah · November 15, 2022, 1:13pm

I don’t think its related to Triton Inference Server since it works with Triton. I believe it’s between CUDA and TensorRT. Should I open an issue TensorRT?

spolisetty · December 1, 2022, 6:55am

Hi,

It could be difficult for us to debug without model and script. Above error can be more related to bindings, please make sure in the input data iterations have correct shapes.

We also recommend you to please try on the latest TensorRT version 8.5.1.

Thank you.