The difference between TF and TRT weights?

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi, I create a network with TRT API, and load tf weights into the TRT network, but the results are different.
I think the weight format difference may lead to the problem. So i have some questions to ask:

(1) for the conv weights, TF uses RSCK ([filter_height, filter_width, input_depth, output_depth]) and TensorRT uses KCRS, so do i need to transpose the weights before feeding to the conv layers? but in the SAMPLEMNISTAPI, there is no tranpose to tf weights for TRT.
for my example:
w1 = weightMap[“tf_conv1”] #RSCK
transpose the tf weights???
conv1 = network.add_convolution(data, 32, (11,41), trt.infer.Weights(w1), b1)

(2) for the IScaleLayer, I set tensorrt.ScaleMode.CHANNEL. The TRT doc says that the channel dimension is assumed to be the third to last dimension, but TRT format is CHW, so do i need to transpose the input of ScaleLayer from CHW to HWC?

(3) for RNN layer, TRT docs says that tf weights format is data_lenhidden_size, while TRT expects hidden_sizedata_len, so should i need to transpose each gate weights of RNN cell?

(4) recently, tf can set data format like channel_first(NCHW) and channel_last(NHWC), so if I train a tf model with data_format=channel_first, are the tf weights format the same as TRT weights format? If so, does it mean I don’t need to convert TF weights for TRT?
Thanks.

2

Hello,

For TensorFlow, the uff parser automatically performs the required format handling.
Note: no matter the model is NCHW or NHWC, remember to register your input blob with NCHW format.

You can find detail information in our document:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#mnist_uff_keyconcepts

3

Hi, what i mean is that I create a network definition from scratch using the python TRT API, and load TF weights to it like:

w1 = weightMap[CONV1_CONVW]
b1 = trt.infer.Weights.empty(trt.infer.DataType.FLOAT)
conv1 = network.add_convolution(data, 32, (11,41), trt.infer.Weights(w1), b1)

and the TF weight format is different from TRT format.

So I wonder whether i need to convert TF weights (w1) to TRT format , then feed to the network.add_convolution API?

4
5
6

Hi @jianxiangm,

did you test transposing the weight format yet? I am doing something similar here: coding the network definition using C++ API and loading the convolutional and bias weights from a trained .h5 file from keras. The C++ code would give me totally different results, even with only one conv layer.

I am also concerned whether the weight format could be an issue and am about to test it out. But i am wondering whether you already tested this?

Thank you very much!

7

The tf weights format is RSCK, but trt weights format is KCRS, so you need to transpose the weights for conv layers.

8

Hi @jianxiangm,

thank you very much for the reply! I transposed the weight format and now my c++ network is able to do inference.

Thanks again for your post in the first place! I didn’t find anything related to weight format in the latest documentation. But from your post, I was able to find documentation for TRT-2.1 or something like that and get a better hold of the issue.