Using beam search with the TensorRT compiled T5 model?

I have been using the code in TensorRT (TensorRT/demo/HuggingFace/T5) that builds decoder and encoder engines from the HuggingFace T5 model. The code works as intended and is very quick for inference.

However, the repo only contains code for performing greedy search with the decoder and I am trying to perform beam search. Are there any plans to update the code with this functionality or are there any pointers/docs for incorporating beam search functionality with a TensorRT model?


We will add beam search support in future releases. Currently, we do not have examples to share.

Thank you.