Description
A bug occurs when using trt to inference batches of images
A clear and concise description of the bug or issue.
Environment
TensorRT Version: TensorRT-7.1.3.4
GPU Type: Tesla V100-PCIE
Nvidia Driver Version: 418.87.00
CUDA Version: 10.2
CUDNN Version: 8.0.2.39
Operating System + Version: Ubuntu18.04
Python Version (if applicable): 3.7.7
TensorFlow Version (if applicable): 2.3.0
PyTorch Version (if applicable): 1.5.1
Baremetal or Container (if container which image + tag):
Relevant Files
Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)
trt_engine:
onnx_file:
Steps To Reproduce
I can successfully inference a single image, but as soon as I loop through a list of images the output of the first image is copied in the output of other images. Below is the related code:
1、to generate dynamic onnx
def transform_to_onnx(weight_file, batch_size, n_classes, IN_IMAGE_H, IN_IMAGE_W):
model = Yolov4(n_classes=n_classes, inference=True)
pretrained_dict = torch.load(weight_file, map_location=torch.device(‘cuda’))
model.load_state_dict(pretrained_dict)
input_names = [“input”]
output_names = [‘boxes’, ‘confs’]
x = torch.randn((1, 3, IN_IMAGE_H, IN_IMAGE_W), requires_grad=True)
onnx_file_name = “yolov4_-1_3_{}_{}_dynamic.onnx”.format(IN_IMAGE_H, IN_IMAGE_W)
dynamic_axes = {“input”: {0: “batch_size”}, “boxes”: {0: “batch_size”}, “confs”: {0: “batch_size”}}
print(‘Export the onnx model …’)
torch.onnx.export(model,x,onnx_file_name,export_params=True,opset_version=11,do_constant_folding=True,
input_names=input_names, output_names=output_names,dynamic_axes=dynamic_axes)
print('Onnx model exporting done')
2、convert to tensorrt engine
def create_optimization_profiles(builder, inputs, batch_size):
# Creates tensorRT optimizations profiles for a given batch size
profiles = [ ]
for inp in inputs:
profile = builder.create_optimization_profile()
shape = inp.shape[1:]
profile.set_shape(inp.name, min=(batch_size, *shape), opt=(batch_size, *shape), max=(batch_size, *shape))
profiles.append(profile)
return profiles
def build_engine(onnx_file_path, engine_file_path, batch_size, verbose=True):
logger = trt.Logger(trt.Logger.VERBOSE) if verbose else trt.Logger()
builder = trt.Builder(logger)
config = builder.create_builder_config()
# Specifies that network should have an explicit batch size (required in tensorRT 7.0.0+)
explicit_batch = [1 << (int)(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)]
network = builder.create_network(*explicit_batch)
parser = trt.OnnxParser(network, logger)
# Define standard settings for tensorRT builder environment
builder.max_workspace_size = 1 << 30
builder.max_batch_size = batch_size
builder.fp16_mode = True
# builder.strict_type_constraints = True
# Parse onnx model
with open(onnx_file_path, ‘rb’) as onnx_model:
if not parser.parse(onnx_model.read()):
print(“ERROR: Failed to parse onnx model.”)
for error in range(parser.num_errors):
print(parser.get_error(error))
return
# Add optimization profiles
inputs = [network.get_input(i) for i in range(network.num_inputs)]
opt_profiles = create_optimization_profiles(builder, inputs, batch_size)
for profile in opt_profiles:
config.add_optimization_profile(profile)
# Explicitly set the the output layer so engine knows where to expect final outputs
last_layer = network.get_layer(network.num_layers - 1)
if not last_layer.get_output(0):
network.mark_output(last_layer.get_output(0))
print(‘Building tensorRT engine…’)
engine = builder.build_engine(network, config)
print(‘Successfully built engine’)
with open(engine_file_path, ‘wb’) as f:
f.write(engine.serialize())
return onnx_file_name
Please include:
- Exact steps/commands to build your repro
- Exact steps/commands to run your repro
- Full traceback of errors encountered
Thanks in advance. I want to inference multiple images successfully. I think this has something to do with context.enqueue. I am not quite sure