# Dynamic engine building and optimization profile **URL:** \ **Category:** TensorRT\ **Created:** [December 29, 2020, 9:16am UTC](https://forums.developer.nvidia.com/t/dynamic-engine-building-and-optimization-profile/164744 "2020-12-29T09:16:41Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![anubhav4sachan](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anubhav4sachan](https://forums.developer.nvidia.com/u/anubhav4sachan)\ **Post date:** [December 29, 2020, 9:16am UTC](https://forums.developer.nvidia.com/t/dynamic-engine-building-and-optimization-profile/164744/1 "2020-12-29T09:16:41Z")
## Description I have exported an ONNX model of SSD based from [PyTorch](https://pytorch.org/hub/nvidia_deeplearningexamples_ssd/). This `.pth` model and `.onnx` model are perfectly working fine. Input is (batch\_size, 3, 300, 300). I have also built the TensorRT engine with dynamic shape (batch\_size being dynamic) such that input layer is (-1, C, H, W). During memory allocation, on host and device, it is done based on engine binding shape i.e. (-1, C, H, W). Since, it is negative, it shows `cuda out of memory` Ways to resolve this (tried and failed): 1. Created optimization profile, and activate it, and then set the binding shapes: The profile has no effect on engine. 2. Manually set binding’s shape to (1, C, H, W). No error, but wrong inference. What am I doing wrong? ## Environment **TensorRT Version** : 7.0 **GPU Type** : Quadro GV100, 32505 MiB **Nvidia Driver Version** : 440.59 **CUDA Version** : 10.2 **CUDNN Version** : **Operating System + Version** : Ubuntu 18.04 **Python Version (if applicable)**: 3.6.9 **TensorFlow Version (if applicable)**: **PyTorch Version (if applicable)**: **Baremetal or Container (if container which image + tag)**: [nvcr.io/nvidia/tensorrt:20.03-py3](http://nvcr.io/nvidia/tensorrt:20.03-py3) ## Steps To Reproduce ```python img is the preprocessed image as descibed by pytorch for the model. ... TRT_LOGGER = trt.Logger(trt.Logger.WARNING) EXPLICIT_BATCH = 1 << (int)(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH) class HostDeviceMem(object): def __init__ (self, host_mem, device_mem): self.host = host_mem self.device = device_mem def __str__ (self): return "Host:\n" + str(self.host) + "\nDevice:\n" + str(self.device) def __repr__ (self): return self. __str__ () def allocate_buffers(engine): inputs = [] outputs = [] bindings = [] stream = cuda.Stream() for binding in engine: size = trt.volume(engine.get_binding_shape(binding)) * -1 * engine.max_batch_size dtype = trt.nptype(engine.get_binding_dtype(binding)) host_mem = cuda.pagelocked_empty(size, dtype) device_mem = cuda.mem_alloc(host_mem.nbytes) bindings.append(int(device_mem)) if engine.binding_is_input(binding): inputs.append(HostDeviceMem(host_mem, device_mem)) else: outputs.append(HostDeviceMem(host_mem, device_mem)) return inputs, outputs, bindings, stream def build_engine_onnx(model_file): with trt.Builder(TRT_LOGGER) as builder, builder.create_builder_config() as config, builder.create_network(EXPLICIT_BATCH) as network, trt.OnnxParser(network, TRT_LOGGER) as parser: builder.max_workspace_size = 1 << 28 builder.max_batch_size = 1 builder.fp16_mode = True #builder.strict_type_constraints = True print('Loading ONNX file from path {}...'.format(model_file)) with open(model_file, 'rb') as model: print('Beginning ONNX file parsing') if not parser.parse(model.read()): print('ERROR: Failed to parse the ONNX file.') for error in range(parser.num_errors): print(parser.get_error(error)) return None # if trt. __version__ [0] >= '7': # # Reshape input to batch size 1 # shape = list(network.get_input(0).shape) # print(shape) # shape[0] = batch_size # network.get_input(0).shape = shape # print(network.get_input(0).shape) profile = builder.create_optimization_profile() profile.set_shape("input", (batch_size, 3, 300, 300), (batch_size, 3, 300, 300), (batch_size, 3, 300, 300)) config.add_optimization_profile(profile) print('Completed parsing of ONNX file') print('Building an engine; this may take a while...') engine = builder.build_engine(network, config) print('Completed creating engine') with open("ssd.engine", 'wb') as f: f.write(engine.serialize()) print('Completed serializing and saving the engine') def read_engine(ename): print("Reading engine: ", ename) with open(ename, 'rb') as f, trt.Runtime(TRT_LOGGER) as runtime: engine = runtime.deserialize_cuda_engine(f.read()) return engine onnx_model_file = "ssd.onnx" ename = ".".join([str(onnx_model_file.split(".")[0]), "engine"]) if not os.path.exists(ename): build_engine_onnx(onnx_model_file) with read_engine(ename) as engine: with engine.create_execution_context() as context: print("before B shape", context.get_binding_shape(0)) context.active_optimization_profile = 0 context.set_binding_shape(0, (batch_size, 3, 300, 300)) print("after B shape", context.get_binding_shape(0)) inputs, outputs, bindings, stream = allocate_buffers(engine) np.copyto(inputs[0].host, img.ravel()) [cuda.memcpy_htod_async(inp.device, inp.host, stream) for inp in inputs] context.execute_async_v2(bindings=bindings, stream_handle=stream.handle) [cuda.memcpy_dtoh_async(out.host, out.device, stream) for out in outputs] stream.synchronize() trt_outs = [out.host for out in outputs] print(trt_outs[0].shape) print(trt_outs[1].shape) ``` ---
**Author:** ![AakankshaS](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aakankshas/32/14047_2.png) [@AakankshaS](https://forums.developer.nvidia.com/u/AakankshaS)\ **Post date:** [December 29, 2020, 3:48pm UTC](https://forums.developer.nvidia.com/t/dynamic-engine-building-and-optimization-profile/164744/3 "2020-12-29T15:48:37Z")
Hi @anubhav4sachan, Request you to share the verbose logs along with your onnx model so that we can assist you better on this. Thanks! ---
**Author:** ![anubhav4sachan](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anubhav4sachan](https://forums.developer.nvidia.com/u/anubhav4sachan)\ **Post date:** [December 29, 2020, 4:25pm UTC](https://forums.developer.nvidia.com/t/dynamic-engine-building-and-optimization-profile/164744/4 "2020-12-29T16:25:02Z")
[Google Drive Link](https://drive.google.com/file/d/1RQysjtezjG_826-croo0O7yvM-bMgOxT/view?usp=sharing) for the `.onnx` model. The inference results are wrong [Log 1, for the given code.] > [@anubhav4sachan](#): > > `size = trt.volume(engine.get_binding_shape(binding)) * -1 * engine.max_batch_size` If the quoted line is changed to `size = trt.volume(engine.get_binding_shape(binding)) * engine.max_batch_size` then there is cuda out of memory error. [Log 2] ### Log 1 of 2 ```auto Loading ONNX file from path ssd.onnx... Beginning ONNX file parsing [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. Completed parsing of ONNX file Building an engine; this may take a while... Completed creating engine Completed serializing and saving the engine Reading engine: ssd.engine before B shape (-1, 3, 300, 300) [TensorRT] WARNING: Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles after B shape (1, 3, 300, 300) (34928,) (707292,) ``` ### Log 2 of 2 ```auto Loading ONNX file from path ssd.onnx... Beginning ONNX file parsing [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. Completed parsing of ONNX file Building an engine; this may take a while... Completed creating engine Completed serializing and saving the engine Reading engine: ssd.engine before B shape (-1, 3, 300, 300) [TensorRT] WARNING: Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles after B shape (1, 3, 300, 300) engine.binding_shape: (-1, 3, 300, 300) size trying to allocate(trt.volume): -270000 Traceback (most recent call last): File "trtssd.py", line 317, in inputs, outputs, bindings, stream = allocate_buffers(engine) File "trtssd.py", line 233, in allocate_buffers host_mem = cuda.pagelocked_empty(size, dtype) pycuda._driver.MemoryError: cuMemHostAlloc failed: out of memory ``` ---
**Author:** ![AakankshaS](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aakankshas/32/14047_2.png) [@AakankshaS](https://forums.developer.nvidia.com/u/AakankshaS)\ **Post date:** [January 15, 2021, 12:08pm UTC](https://forums.developer.nvidia.com/t/dynamic-engine-building-and-optimization-profile/164744/5 "2021-01-15T12:08:49Z")
Hi @anubhav4sachan, I could not reproduce the issue. Can you please running your model with TRT trtexec command with the latest release. [https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec](https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec) Thanks! ---
**Author:** ![brad.steiner1](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@brad.steiner1](https://forums.developer.nvidia.com/u/brad.steiner1)\ **Post date:** [July 18, 2023, 3:45pm UTC](https://forums.developer.nvidia.com/t/dynamic-engine-building-and-optimization-profile/164744/6 "2023-07-18T15:45:43Z")
@anubhav4sachan were you ever able to resolve this?