the problem shown as the attached file that when we loop our inference for several times, each image per loop, it always shows the result of the first loop.
Like, these two images are different result[ like OK or NG], but the engine will infer the same [ like NG].
And if reload the engine, the result may get the same different result[ like OK].
TensorRT Version: 188.8.131.52
GPU Type: RTX5000
Nvidia Driver Version: 515.65
CUDA Version: 11.7
Operating System + Version: ubuntu 22.02
Python Version (if applicable): python3.8
The env was [download] by this way
docker run --runtime=nvidia --gpus '"'device=3'"' -it --rm --entrypoint "" nvcr.io/nvidia/tensorrt:22.05-py3 /bin/bash
and the result is
[I] The logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.
the trt loger command is
### Load TensorRT engine trt_logger = trt.Logger(trt.Logger.INFO) runtime = trt.Runtime(trt_logger) with open(ENGINE_PATH, "rb") as f: engine = runtime.deserialize_cuda_engine(f.read())
And infer is
def infer(img): bindings = [int(devide_in), int(devide_out)] np.copyto(host_in, img.ravel()) cuda.memcpy_htod_async(devide_in, host_in, stream) context.execute_async(bindings=bindings, stream_handle=stream.handle) cuda.memcpy_dtoh_async(host_out, devide_out, stream) stream.synchronize() return host_out
I try to use init() and pop() to avoid the existing trt value, but it was uselessful.
cuda.init() device = cuda.Device(0) ctx = device.make_context() ctx.pop()
How to avoid the logger return the existing value? It puzzled me for a long time!