Zeros-like tensor that has dynamic shape in TensorRT(-LLM)

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I have an operation where, at runtime, I need to replace an input tensor with a dynamic shape (1, -1) with a zero-like tensor that also has a dynamic shape (1, -1) in TensorRT-LLM. my first try is something like this:

dropped_text = constant(np.ascontiguousarray(np.zeros([1, max_len], dtype=trt_dtype_to_np(trt.int32))))

so in this case, it’s quite obvious that numpy is not going to work with (max_len=-1) in this case, so I’m looking for a solution.

2

Hi @saeed.masoomi2013 Please take a look at the documentation on Dynamic Shapes in TRT-LLM Developer Guide :: NVIDIA Deep Learning TensorRT Documentation, Sophie