Hi,

I’m using TensorRT 4.0 with LSTM on our Tesla cluster. But I encounter the following problem.

We are using kLSTM denoted in the following equation

i[t] := sigmoid(W[i].X[t] + R[i].H[t-1] + Wb[i] + Rb[i])

f[t] := sigmoid(W[f].X[t] + R[f].H[t-1] + Wb[f] + Rb[f])

o[t] := sigmoid(W[o].X[t] + R[o].H[t-1] + Wb[o] + Rb[o])

c[t] := tanh(W[c].X[t] + R[c].H[t-1] + Wb[c] + Rb[c])

C[t] := f[t]*C[t-1] + i[t]*c[t]

H[t] := o[t]*tanh(C[t])

But there’s a small difference as we are using ReLU instead of tanh in

c[t] := ReLU(W[c].X[t] + R[c].H[t-1] + Wb[c] + Rb[c])

We found the original equation in this page

https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/c_api/namespacenvinfer1.html#ace7b656a1c0537ea0edd17cf61121200

Some papers including Baidu’s DeepSpeech 2 are using ReLU instead of tanh. (You can find code in their PaddlePaddle github repo).

So my questions are

- How can we use ReLU instead of tanh in the equation of c[t] in TensorRT 4.x?
- If it’s impossible in 4.x, is there any plan or future version for TensorRT to support this customization?
- If TRT won’t support this customization, how can we implement it?

Thank you