Hi , i optimized my model (tracking) with tensorrt , the input shape is [[?,128,64,3]] and output shape is [?,128], i use the jetson xavier development kits, when i run the inference every time i detect a new person and add it to the tracker the display windows is blocked because it build a new tensorrt engine , so it influence time speed , how can i resolve this problem ?
Building a new TensorRT engine for net/TRTEngineOp_1 input shapes: [[1,128,64,3]]
Building a new TensorRT engine for net/TRTEngineOp_1 input shapes: [[2,128,64,3]]
Building a new TensorRT engine for net/TRTEngineOp_1 input shapes: [[3,128,64,3]]
As they say here:
Dynamic mode allows you to have unknown shapes in your model, despite the fact that TensorRT requires all shapes to be fully defined. In this mode, TF-TRT creates a new TensorRT engine for each unique input shape that is supplied to the model. For example, you may have an image classification network that works on images of any size where the input placeholder has the shape [?, ?, ?, 3]. If you were to first send a batch of images to the model with shape [8, 224, 224, 3], a new TensorRT engine will be created that is optimized for those dimensions. Since the engine will have to be built at this time, this first batch will take longer to execute than usual. If you later send more images with the same shape of [8, 224, 224, 3], the previously built engine will be used immediately with no additional overhead. If you instead send a batch with a different shape, a new engine would have to be created for that shape. The argument maximum_cached_engines can be used to control how many engines will be stored at a time, for each individual TRTEngineOp in the graph.
i used batch_size=1 it’s work fine but when i changed the size of batch i get the same problem , i need to use batch !!