Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:11:52] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:11:52] [V] [TRT] Original: 18 layers [05/23/2020-11:11:52] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:11:52] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:11:52] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:11:52] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:11:52] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:11:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:11:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:11:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:11:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:11:52] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:11:52] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:11:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:11:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:11:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:11:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:11:52] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:11:52] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:11:52] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:11:52] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:11:52] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:11:52] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:11:52] [V] [TRT] Graph construction and optimization completed in 0.00274702 seconds. [05/23/2020-11:11:54] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:54] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:11:54] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:54] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:11:54] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:11:54] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:11:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:11:54] [V] [TRT] Tactic: 1825138533642645384 time 0.09216 [05/23/2020-11:11:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:11:54] [V] [TRT] Tactic: 3915320020053085238 time 0.091136 [05/23/2020-11:11:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:11:54] [V] [TRT] Tactic: 6808617066150061604 time 0.058368 [05/23/2020-11:11:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:11:54] [V] [TRT] Tactic: -8060443123034038864 time 0.062464 [05/23/2020-11:11:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:11:54] [V] [TRT] Tactic: -4420849921117327522 time 0.070656 [05/23/2020-11:11:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:11:54] [V] [TRT] Tactic: -3946921629105938337 time 0.084 [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.058368 [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 time 0.044032 [05/23/2020-11:11:54] [V] [TRT] Tactic: 1 time 0.065536 [05/23/2020-11:11:54] [V] [TRT] Tactic: 2 time 0.09216 [05/23/2020-11:11:54] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:11:54] [V] [TRT] Tactic: 5 time 0.176128 [05/23/2020-11:11:54] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0.044032 [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:11:54] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:11:54] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:11:54] [V] [TRT] [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:11:54] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:11:54] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:11:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:11:54] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:11:54] [V] [TRT] Tactic: 2 time 0.008096 [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:11:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:11:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:11:54] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:11:54] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:11:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:11:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:11:54] [V] [TRT] Tactic: 1825138533642645384 time 0.263168 [05/23/2020-11:11:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:11:54] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:11:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:11:54] [V] [TRT] Tactic: 6808617066150061604 time 0.160768 [05/23/2020-11:11:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: -8060443123034038864 time 0.172032 [05/23/2020-11:11:55] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: -4420849921117327522 time 0.190464 [05/23/2020-11:11:55] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.160768 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:11:55] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:11:55] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:11:55] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:11:55] [V] [TRT] Tactic: 5 time 0.357376 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:11:55] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:11:55] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:11:55] [V] [TRT] [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:11:55] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:11:55] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:11:55] [V] [TRT] Tactic: 1 time 0.007136 [05/23/2020-11:11:55] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 1 Time: 0.007136 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] Formats and tactics selection completed in 0.599252 seconds. [05/23/2020-11:11:55] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:11:55] [V] [TRT] Block size 1073741824 [05/23/2020-11:11:55] [V] [TRT] Block size 153600 [05/23/2020-11:11:55] [V] [TRT] Block size 153600 [05/23/2020-11:11:55] [V] [TRT] Block size 2048 [05/23/2020-11:11:55] [V] [TRT] Block size 2048 [05/23/2020-11:11:55] [V] [TRT] Block size 2048 [05/23/2020-11:11:55] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:11:55] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:11:55] [V] [TRT] Engine generation completed in 2.5588 seconds. [05/23/2020-11:11:55] [V] [TRT] Engine Layer Information: [05/23/2020-11:11:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:11:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:11:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:11:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:11:55] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:11:55] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:11:55] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:11:55] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:11:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:11:55] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:11:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:11:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:11:55] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:11:55] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:11:55] [V] [TRT] Original: 48 layers [05/23/2020-11:11:55] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:11:55] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:11:55] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:11:55] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:11:55] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:11:55] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:11:55] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:11:55] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:11:55] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:11:55] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:11:55] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:11:55] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:11:55] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:11:55] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:11:55] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:11:55] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:11:55] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:11:55] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:11:55] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:11:55] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:11:55] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:11:55] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:11:55] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:11:55] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:11:55] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:11:55] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:11:55] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:11:55] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:11:55] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:11:55] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:11:55] [V] [TRT] Graph construction and optimization completed in 0.0220399 seconds. [05/23/2020-11:11:55] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:11:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:11:55] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:11:55] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:11:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:11:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: 2842488832350522458 time 0.017344 [05/23/2020-11:11:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:11:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:11:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:11:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:11:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:11:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:11:55] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:11:55] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:11:55] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:11:55] [V] [TRT] Tactic: 4 time 1.6209 [05/23/2020-11:11:55] [V] [TRT] Tactic: 5 time 0.038848 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:11:55] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:11:55] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:11:55] [V] [TRT] [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:11:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:11:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.005184 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.005184 [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:11:55] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:11:55] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:11:55] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:11:55] [V] [TRT] Tactic: -32 time 0.00928 [05/23/2020-11:11:55] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:11:55] [V] [TRT] Tactic: -128 time 0.009216 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 128 Time: 0.008192 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:11:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:11:55] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:11:55] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:11:55] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:11:55] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:11:55] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:11:55] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:11:56] [V] [TRT] Tactic: 128 time 0.006208 [05/23/2020-11:11:56] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Tactic: 512 time 0.006176 [05/23/2020-11:11:56] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:11:56] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:11:56] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:11:56] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:11:56] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:11:56] [V] [TRT] Tactic: 6 time 0.104448 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:11:56] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:11:56] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:11:56] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:11:56] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] Formats and tactics selection completed in 1.27573 seconds. [05/23/2020-11:11:56] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:11:56] [V] [TRT] Block size 1073741824 [05/23/2020-11:11:56] [V] [TRT] Block size 38400 [05/23/2020-11:11:56] [V] [TRT] Block size 38400 [05/23/2020-11:11:56] [V] [TRT] Block size 4608 [05/23/2020-11:11:56] [V] [TRT] Block size 2560 [05/23/2020-11:11:56] [V] [TRT] Block size 1024 [05/23/2020-11:11:56] [V] [TRT] Block size 1024 [05/23/2020-11:11:56] [V] [TRT] Block size 0 [05/23/2020-11:11:56] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:11:56] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:11:56] [V] [TRT] Engine generation completed in 1.32383 seconds. [05/23/2020-11:11:56] [V] [TRT] Engine Layer Information: [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:11:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:11:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:11:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:11:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:11:56] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:11:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:11:56] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:11:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:11:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:11:56] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:11:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:11:56] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:11:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:11:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:11:56] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:11:56] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:11:56] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:11:56] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 256, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:11:56] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:11:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:11:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:11:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:11:56] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:11:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:11:56] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:11:56] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:11:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:11:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:11:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:11:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:11:56] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:11:56] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:11:56] [V] [TRT] Original: 12 layers [05/23/2020-11:11:56] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:11:56] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:11:56] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:11:56] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:11:56] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:11:56] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:11:56] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:11:56] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:11:56] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:11:56] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:11:56] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:11:56] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:11:56] [V] [TRT] Graph construction and optimization completed in 0.00517822 seconds. [05/23/2020-11:11:56] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:11:56] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:11:56] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:11:56] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:11:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:11:56] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:11:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:11:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:11:56] [V] [TRT] Formats and tactics selection completed in 0.0670209 seconds. [05/23/2020-11:11:56] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:11:56] [V] [TRT] Block size 1073741824 [05/23/2020-11:11:56] [V] [TRT] Block size 512 [05/23/2020-11:11:56] [V] [TRT] Block size 512 [05/23/2020-11:11:56] [V] [TRT] Block size 512 [05/23/2020-11:11:56] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:11:56] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:11:56] [V] [TRT] Engine generation completed in 0.0840902 seconds. [05/23/2020-11:11:56] [V] [TRT] Engine Layer Information: [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:11:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:11:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:11:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:11:56] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:11:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:11:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:11:56] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:11:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:11:56] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:11:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread2 load float count:3834 thread4 load float count:3834 thread7 load float count:3834 thread6 load float count:3834 thread5 load float count:3834 thread8 load float count:3834 thread9 load float count:3834 thread10 load float count:3834 thread12 load float count:3834 thread11 load float count:3834 thread14 load float count:3834 thread13 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread19 load float count:3834 thread18 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 16 finish thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 thread 19 finish The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish thread 12 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:12:12] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:12:12] [V] [TRT] Original: 18 layers [05/23/2020-11:12:12] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:12:12] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:12:12] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:12:12] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:12:12] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:12:12] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:12] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:12] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:12] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:12] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:12:12] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:12:12] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:12] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:12] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:12] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:12] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:12:12] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:12:12] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:12:12] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:12:12] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:12:12] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:12:12] [V] [TRT] Graph construction and optimization completed in 0.00272998 seconds. [05/23/2020-11:12:14] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:12:14] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:12:14] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: -8060443123034038864 time 0.057344 [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: -4420849921117327522 time 0.065536 [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: -3946921629105938337 time 0.078848 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.041984 [05/23/2020-11:12:14] [V] [TRT] Tactic: 1 time 0.063488 [05/23/2020-11:12:14] [V] [TRT] Tactic: 2 time 0.08704 [05/23/2020-11:12:14] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:12:14] [V] [TRT] Tactic: 5 time 0.175104 [05/23/2020-11:12:14] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.041984 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:12:14] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:14] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:12:14] [V] [TRT] [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:12:14] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:14] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:12:14] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:14] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:12:14] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:12:14] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: 1825138533642645384 time 0.264192 [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: 6808617066150061604 time 0.15872 [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: -4420849921117327522 time 0.145408 [05/23/2020-11:12:14] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:12:14] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.145408 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:12:14] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:12:14] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:12:14] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:12:14] [V] [TRT] Tactic: 5 time 0.356448 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:12:14] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:14] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:12:14] [V] [TRT] [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:14] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:14] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:14] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:14] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:12:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:12:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:12:14] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] Formats and tactics selection completed in 0.83173 seconds. [05/23/2020-11:12:15] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:12:15] [V] [TRT] Block size 1073741824 [05/23/2020-11:12:15] [V] [TRT] Block size 153600 [05/23/2020-11:12:15] [V] [TRT] Block size 153600 [05/23/2020-11:12:15] [V] [TRT] Block size 2048 [05/23/2020-11:12:15] [V] [TRT] Block size 2048 [05/23/2020-11:12:15] [V] [TRT] Block size 2048 [05/23/2020-11:12:15] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:12:15] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:12:15] [V] [TRT] Engine generation completed in 2.58096 seconds. [05/23/2020-11:12:15] [V] [TRT] Engine Layer Information: [05/23/2020-11:12:15] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:12:15] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:15] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:12:15] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:12:15] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:12:15] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:12:15] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:12:15] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:12:15] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:12:15] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:12:15] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:12:15] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:12:15] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:12:15] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:12:15] [V] [TRT] Original: 48 layers [05/23/2020-11:12:15] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:12:15] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:12:15] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:12:15] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:15] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:15] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:15] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:15] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:12:15] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:12:15] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:12:15] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:12:15] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:12:15] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:12:15] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:12:15] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:12:15] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:12:15] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:12:15] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:12:15] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:12:15] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:12:15] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:12:15] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:12:15] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:12:15] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:12:15] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:12:15] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:12:15] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:12:15] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:12:15] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:12:15] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:12:15] [V] [TRT] Graph construction and optimization completed in 0.0156275 seconds. [05/23/2020-11:12:15] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:12:15] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:12:15] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:12:15] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:12:15] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:12:15] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:12:15] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:12:15] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:12:15] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:12:15] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:12:15] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:12:15] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:12:15] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:12:15] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:12:15] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:12:15] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:12:15] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:12:15] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:12:15] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:12:15] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:12:15] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 time 0.01024 [05/23/2020-11:12:15] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:12:15] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:12:15] [V] [TRT] Tactic: 4 time 1.62195 [05/23/2020-11:12:15] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0.01024 [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:12:15] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:15] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:12:15] [V] [TRT] [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:15] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:15] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:15] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:15] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:12:15] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:12:15] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:12:15] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:12:15] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:12:15] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:12:15] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:12:15] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:12:15] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:12:15] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:12:15] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:12:15] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:12:15] [V] [TRT] Tactic: -128 time 0.007264 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:12:15] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:15] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:12:15] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:12:15] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:12:15] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:12:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:12:15] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:15] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:15] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:12:16] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:12:16] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:12:16] [V] [TRT] Formats and tactics selection completed in 1.17439 seconds. [05/23/2020-11:12:16] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:12:16] [V] [TRT] Block size 1073741824 [05/23/2020-11:12:16] [V] [TRT] Block size 38400 [05/23/2020-11:12:16] [V] [TRT] Block size 38400 [05/23/2020-11:12:16] [V] [TRT] Block size 4608 [05/23/2020-11:12:16] [V] [TRT] Block size 2560 [05/23/2020-11:12:16] [V] [TRT] Block size 1024 [05/23/2020-11:12:16] [V] [TRT] Block size 1024 [05/23/2020-11:12:16] [V] [TRT] Block size 0 [05/23/2020-11:12:16] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:12:16] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:12:16] [V] [TRT] Engine generation completed in 1.22166 seconds. [05/23/2020-11:12:16] [V] [TRT] Engine Layer Information: [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:12:16] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:16] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:16] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:16] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:12:16] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:12:16] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:12:16] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:12:16] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:12:16] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:12:16] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:12:16] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:12:16] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:12:16] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:12:16] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:12:16] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:12:16] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:12:16] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:12:16] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:12:16] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:12:16] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:12:16] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:12:16] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:12:16] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:12:16] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:12:16] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:12:16] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:12:16] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:12:16] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:12:16] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:12:16] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:12:16] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:12:16] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:12:16] [V] [TRT] Original: 12 layers [05/23/2020-11:12:16] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:12:16] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:12:16] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:12:16] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:12:16] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:12:16] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:12:16] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:12:16] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:12:16] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:12:16] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:12:16] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:12:16] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:12:16] [V] [TRT] Graph construction and optimization completed in 0.00495342 seconds. [05/23/2020-11:12:16] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:12:16] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:12:16] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:16] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:12:16] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:16] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:12:16] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:16] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:16] [V] [TRT] Formats and tactics selection completed in 0.347969 seconds. [05/23/2020-11:12:16] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:12:16] [V] [TRT] Block size 1073741824 [05/23/2020-11:12:16] [V] [TRT] Block size 512 [05/23/2020-11:12:16] [V] [TRT] Block size 512 [05/23/2020-11:12:16] [V] [TRT] Block size 512 [05/23/2020-11:12:16] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:12:16] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:12:16] [V] [TRT] Engine generation completed in 0.370648 seconds. [05/23/2020-11:12:16] [V] [TRT] Engine Layer Information: [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:12:16] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:12:16] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:16] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:16] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:12:16] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:16] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:16] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:12:16] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:12:16] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:16] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:17] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:17] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:17] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:17] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:17] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:17] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:17] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread2 load float count:3834 thread0 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread6 load float count:3834 thread5 load float count:3834 thread4 load float count:3834 thread7 load float count:3834 thread9 load float count:3834 thread8 load float count:3834 thread12 load float count:3834 thread10 load float count:3834 thread11 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread16 load float count:3834 thread18 load float count:3834 thread15 load float count:3834 thread17 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 12 finish The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 19 finish thread 18 finish The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 thread 11 finish The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 7 finish thread 9 finish thread 14 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:12:33] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:12:33] [V] [TRT] Original: 18 layers [05/23/2020-11:12:33] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:12:33] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:12:33] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:12:33] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:12:33] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:12:33] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:33] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:33] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:33] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:33] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:12:33] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:12:33] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:33] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:33] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:33] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:33] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:12:33] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:12:33] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:12:33] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:12:33] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:12:33] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:12:33] [V] [TRT] Graph construction and optimization completed in 0.00255056 seconds. [05/23/2020-11:12:35] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:12:35] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:12:35] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: 1825138533642645384 time 0.09216 [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: 3915320020053085238 time 0.091136 [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: 6808617066150061604 time 0.058368 [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: -8060443123034038864 time 0.063488 [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: -4420849921117327522 time 0.07168 [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: -3946921629105938337 time 0.086016 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.058368 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.045024 [05/23/2020-11:12:35] [V] [TRT] Tactic: 1 time 0.068608 [05/23/2020-11:12:35] [V] [TRT] Tactic: 2 time 0.094208 [05/23/2020-11:12:35] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:12:35] [V] [TRT] Tactic: 5 time 0.185344 [05/23/2020-11:12:35] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.045024 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:12:35] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:35] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:12:35] [V] [TRT] [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.0072 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.0072 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:12:35] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:35] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:12:35] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:35] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:12:35] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:12:35] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: 1825138533642645384 time 0.26112 [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: 3915320020053085238 time 0.259072 [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: 6808617066150061604 time 0.151552 [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: -8060443123034038864 time 0.162816 [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: -4420849921117327522 time 0.145408 [05/23/2020-11:12:35] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:12:35] [V] [TRT] Tactic: -3946921629105938337 time 0.183296 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.145408 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.09728 [05/23/2020-11:12:35] [V] [TRT] Tactic: 1 time 0.15872 [05/23/2020-11:12:35] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:12:35] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:12:35] [V] [TRT] Tactic: 5 time 0.356352 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.09728 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:12:35] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:35] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:12:35] [V] [TRT] [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:35] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:35] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:35] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:35] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.007232 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.007232 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] Formats and tactics selection completed in 0.6178 seconds. [05/23/2020-11:12:35] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:12:35] [V] [TRT] Block size 1073741824 [05/23/2020-11:12:35] [V] [TRT] Block size 153600 [05/23/2020-11:12:35] [V] [TRT] Block size 153600 [05/23/2020-11:12:35] [V] [TRT] Block size 2048 [05/23/2020-11:12:35] [V] [TRT] Block size 2048 [05/23/2020-11:12:35] [V] [TRT] Block size 2048 [05/23/2020-11:12:35] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:12:35] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:12:35] [V] [TRT] Engine generation completed in 2.57044 seconds. [05/23/2020-11:12:35] [V] [TRT] Engine Layer Information: [05/23/2020-11:12:35] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:12:35] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:35] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:12:35] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:12:35] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:12:35] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:12:35] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:12:35] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:12:35] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:12:35] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:12:35] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:12:35] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:12:35] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:12:35] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:12:35] [V] [TRT] Original: 48 layers [05/23/2020-11:12:35] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:12:35] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:12:35] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:12:35] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:35] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:35] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:35] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:35] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:12:35] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:12:35] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:12:35] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:12:35] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:12:35] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:12:35] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:12:35] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:12:35] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:12:35] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:12:35] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:12:35] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:12:35] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:12:35] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:12:35] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:12:35] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:12:35] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:12:35] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:12:35] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:12:35] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:12:35] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:12:35] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:12:35] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:12:35] [V] [TRT] Graph construction and optimization completed in 0.0230157 seconds. [05/23/2020-11:12:35] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:12:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:35] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:12:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:12:36] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:12:36] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:12:36] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:12:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:12:36] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:12:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:12:36] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:12:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:12:36] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:12:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:12:36] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:12:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:12:36] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:12:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:12:36] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:12:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:12:36] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:12:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:12:36] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.01024 [05/23/2020-11:12:36] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:12:36] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:12:36] [V] [TRT] Tactic: 4 time 1.61382 [05/23/2020-11:12:36] [V] [TRT] Tactic: 5 time 0.039936 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.01024 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:12:36] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:36] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:12:36] [V] [TRT] [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:36] [V] [TRT] Tactic: 1 time 0.006176 [05/23/2020-11:12:36] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 1 Time: 0.006176 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:36] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:12:36] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:12:36] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:12:36] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:12:36] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:12:36] [V] [TRT] Tactic: -64 time 0.008224 [05/23/2020-11:12:36] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 256 Time: 0.007168 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:12:36] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:12:36] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Tactic: 3 time 0.009216 [05/23/2020-11:12:36] [V] [TRT] Tactic: 6 time 0.050176 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:12:36] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:12:36] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:12:36] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:12:36] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:12:36] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:12:36] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:36] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:12:36] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:12:36] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:36] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:36] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:12:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:12:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:12:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:12:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:12:37] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:12:37] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:12:37] [V] [TRT] Formats and tactics selection completed in 1.23801 seconds. [05/23/2020-11:12:37] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:12:37] [V] [TRT] Block size 1073741824 [05/23/2020-11:12:37] [V] [TRT] Block size 38400 [05/23/2020-11:12:37] [V] [TRT] Block size 38400 [05/23/2020-11:12:37] [V] [TRT] Block size 4608 [05/23/2020-11:12:37] [V] [TRT] Block size 2560 [05/23/2020-11:12:37] [V] [TRT] Block size 1024 [05/23/2020-11:12:37] [V] [TRT] Block size 1024 [05/23/2020-11:12:37] [V] [TRT] Block size 0 [05/23/2020-11:12:37] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:12:37] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:12:37] [V] [TRT] Engine generation completed in 1.3028 seconds. [05/23/2020-11:12:37] [V] [TRT] Engine Layer Information: [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:12:37] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:37] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:37] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:12:37] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:12:37] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:12:37] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:12:37] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:12:37] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:12:37] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:12:37] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:12:37] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:12:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:12:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:12:37] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:12:37] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:12:37] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:12:37] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 256, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:12:37] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:12:37] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:12:37] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:12:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:12:37] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:12:37] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:12:37] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:12:37] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:12:37] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:12:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:12:37] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:12:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:12:37] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:12:37] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:12:37] [V] [TRT] Original: 12 layers [05/23/2020-11:12:37] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:12:37] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:12:37] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:12:37] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:12:37] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:12:37] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:12:37] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:12:37] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:12:37] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:12:37] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:12:37] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:12:37] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:12:37] [V] [TRT] Graph construction and optimization completed in 0.00655484 seconds. [05/23/2020-11:12:37] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:12:37] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:12:37] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:12:37] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 512 Time: 0.006144 [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:37] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:12:37] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:12:37] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:12:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:37] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:12:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:37] [V] [TRT] Formats and tactics selection completed in 0.0685238 seconds. [05/23/2020-11:12:37] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:12:37] [V] [TRT] Block size 1073741824 [05/23/2020-11:12:37] [V] [TRT] Block size 512 [05/23/2020-11:12:37] [V] [TRT] Block size 512 [05/23/2020-11:12:37] [V] [TRT] Block size 512 [05/23/2020-11:12:37] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:12:37] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:12:37] [V] [TRT] Engine generation completed in 0.354783 seconds. [05/23/2020-11:12:37] [V] [TRT] Engine Layer Information: [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:12:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:12:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:37] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 512, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:12:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:37] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:12:37] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:12:37] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread1 load float count:3834 thread2 load float count:3834 thread3 load float count:3834 thread4 load float count:3834 thread6 load float count:3834 thread7 load float count:3834 thread5 load float count:3834 thread9 load float count:3834 thread8 load float count:3834 thread10 load float count:3834 thread11 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 [05/23/2020-11:12:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:12:37] [E] [TRT] FAILED_EXECUTION: std::exception stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread stop token triggered at step: 327, batch_id: 0, 0.999942 18 finish The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish The output sequence length is 1836 thread 10 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:12:52] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:12:52] [V] [TRT] Original: 18 layers [05/23/2020-11:12:52] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:12:52] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:12:52] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:12:52] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:12:52] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:12:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:52] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:12:52] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:12:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:52] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:12:52] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:12:52] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:12:52] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:12:52] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:12:52] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:12:52] [V] [TRT] Graph construction and optimization completed in 0.00279423 seconds. [05/23/2020-11:12:54] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:12:54] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:12:54] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: -8060443123034038864 time 0.057344 [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: -4420849921117327522 time 0.065536 [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: -3946921629105938337 time 0.077856 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.041984 [05/23/2020-11:12:54] [V] [TRT] Tactic: 1 time 0.063488 [05/23/2020-11:12:54] [V] [TRT] Tactic: 2 time 0.086048 [05/23/2020-11:12:54] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:12:54] [V] [TRT] Tactic: 5 time 0.168992 [05/23/2020-11:12:54] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.041984 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:12:54] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:54] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:12:54] [V] [TRT] [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:12:54] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:54] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:12:54] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:54] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.006176 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006176 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:12:54] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:12:54] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: 1825138533642645384 time 0.262144 [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: 3915320020053085238 time 0.260096 [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: 6808617066150061604 time 0.152576 [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: -8060443123034038864 time 0.162816 [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: -4420849921117327522 time 0.145408 [05/23/2020-11:12:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:12:54] [V] [TRT] Tactic: -3946921629105938337 time 0.183296 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.145408 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.09728 [05/23/2020-11:12:54] [V] [TRT] Tactic: 1 time 0.15872 [05/23/2020-11:12:54] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:12:54] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:12:54] [V] [TRT] Tactic: 5 time 0.352256 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.09728 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:12:54] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:54] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:12:54] [V] [TRT] [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:54] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:54] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:54] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:54] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:12:54] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:12:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:54] [V] [TRT] Formats and tactics selection completed in 0.604107 seconds. [05/23/2020-11:12:54] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:12:54] [V] [TRT] Block size 1073741824 [05/23/2020-11:12:54] [V] [TRT] Block size 153600 [05/23/2020-11:12:54] [V] [TRT] Block size 153600 [05/23/2020-11:12:54] [V] [TRT] Block size 2048 [05/23/2020-11:12:54] [V] [TRT] Block size 2048 [05/23/2020-11:12:54] [V] [TRT] Block size 2048 [05/23/2020-11:12:54] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:12:54] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:12:54] [V] [TRT] Engine generation completed in 2.64743 seconds. [05/23/2020-11:12:54] [V] [TRT] Engine Layer Information: [05/23/2020-11:12:54] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:12:54] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:54] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:12:54] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:12:54] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:12:54] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:12:54] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:12:54] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:12:54] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:12:54] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:12:54] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:12:54] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:12:54] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:12:54] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:12:54] [V] [TRT] Original: 48 layers [05/23/2020-11:12:54] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:12:54] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:12:54] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:12:54] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:54] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:54] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:12:54] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:12:54] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:12:54] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:12:54] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:12:55] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:12:55] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:12:55] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:12:55] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:12:55] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:12:55] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:12:55] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:12:55] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:12:55] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:12:55] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:12:55] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:12:55] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:12:55] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:12:55] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:12:55] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:12:55] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:12:55] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:12:55] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:12:55] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:12:55] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:12:55] [V] [TRT] Graph construction and optimization completed in 0.0215529 seconds. [05/23/2020-11:12:55] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:12:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:12:55] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:12:55] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:12:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:12:55] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:12:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:12:55] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:12:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:12:55] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:12:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:12:55] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:12:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:12:55] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:12:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:12:55] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:12:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:12:55] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:12:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:12:55] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:12:55] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:12:55] [V] [TRT] Tactic: 2 time 0.01536 [05/23/2020-11:12:55] [V] [TRT] Tactic: 4 time 1.61075 [05/23/2020-11:12:55] [V] [TRT] Tactic: 5 time 0.036832 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:12:55] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:12:55] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:12:55] [V] [TRT] [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:12:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:12:55] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:12:55] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:12:55] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:12:55] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:12:55] [V] [TRT] Tactic: -64 time 0.009216 [05/23/2020-11:12:55] [V] [TRT] Tactic: -128 time 0.009216 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 128 Time: 0.008192 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:12:55] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:12:55] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:12:55] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:12:55] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:12:55] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:12:55] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:12:55] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:12:55] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:12:55] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:12:55] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:12:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:12:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:12:55] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:12:56] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:12:56] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:12:56] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:12:56] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:12:56] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:12:56] [V] [TRT] Formats and tactics selection completed in 1.27315 seconds. [05/23/2020-11:12:56] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:12:56] [V] [TRT] Block size 1073741824 [05/23/2020-11:12:56] [V] [TRT] Block size 38400 [05/23/2020-11:12:56] [V] [TRT] Block size 38400 [05/23/2020-11:12:56] [V] [TRT] Block size 4608 [05/23/2020-11:12:56] [V] [TRT] Block size 2560 [05/23/2020-11:12:56] [V] [TRT] Block size 1024 [05/23/2020-11:12:56] [V] [TRT] Block size 1024 [05/23/2020-11:12:56] [V] [TRT] Block size 0 [05/23/2020-11:12:56] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:12:56] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:12:56] [V] [TRT] Engine generation completed in 1.32294 seconds. [05/23/2020-11:12:56] [V] [TRT] Engine Layer Information: [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:12:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:12:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:12:56] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:12:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:12:56] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:12:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:12:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:12:56] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:12:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:12:56] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:12:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:12:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:12:56] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:12:56] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:12:56] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:12:56] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 256, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:12:56] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:12:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:12:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:12:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:12:56] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:12:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:12:56] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:12:56] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:12:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:12:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:12:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:12:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:12:56] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:12:56] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:12:56] [V] [TRT] Original: 12 layers [05/23/2020-11:12:56] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:12:56] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:12:56] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:12:56] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:12:56] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:12:56] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:12:56] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:12:56] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:12:56] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:12:56] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:12:56] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:12:56] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:12:56] [V] [TRT] Graph construction and optimization completed in 0.00543156 seconds. [05/23/2020-11:12:56] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:12:56] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:12:56] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:12:56] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:12:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:12:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:12:56] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:12:56] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:12:56] [V] [TRT] Tactic: 512 time 0.006176 [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:12:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:56] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:12:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:12:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:12:56] [V] [TRT] Formats and tactics selection completed in 0.069182 seconds. [05/23/2020-11:12:56] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:12:56] [V] [TRT] Block size 1073741824 [05/23/2020-11:12:56] [V] [TRT] Block size 512 [05/23/2020-11:12:56] [V] [TRT] Block size 512 [05/23/2020-11:12:56] [V] [TRT] Block size 512 [05/23/2020-11:12:56] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:12:56] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:12:56] [V] [TRT] Engine generation completed in 0.0907248 seconds. [05/23/2020-11:12:56] [V] [TRT] Engine Layer Information: [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:12:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:12:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:56] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:12:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:12:56] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 256, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:12:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:12:56] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:12:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread1 load float count:3834 thread2 load float count:3834 thread3 load float count:3834 thread4 load float count:3834 thread6 load float count:3834 thread5 load float count:3834 thread7 load float count:3834 thread9 load float count:3834 thread8 load float count:3834 thread10 load float count:3834 thread11 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 18 finish The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 thread 0 finish The output sequence length is 654 The output sequence length is 654 thread 5 finish thread 1 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:13:10] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:13:10] [V] [TRT] Original: 18 layers [05/23/2020-11:13:10] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:13:10] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:13:10] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:13:10] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:13:10] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:13:10] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:10] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:10] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:10] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:10] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:13:10] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:13:10] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:10] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:10] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:10] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:10] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:13:10] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:13:10] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:13:10] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:13:10] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:13:10] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:13:10] [V] [TRT] Graph construction and optimization completed in 0.00241245 seconds. [05/23/2020-11:13:11] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:13:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:13:11] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:11] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:11] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:13:11] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:11] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:13:12] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:13:12] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: 1825138533642645384 time 0.091136 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: 3915320020053085238 time 0.090112 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: 6808617066150061604 time 0.058368 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: -8060443123034038864 time 0.063488 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: -4420849921117327522 time 0.070656 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: -3946921629105938337 time 0.085984 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.058368 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.044032 [05/23/2020-11:13:12] [V] [TRT] Tactic: 1 time 0.067584 [05/23/2020-11:13:12] [V] [TRT] Tactic: 2 time 0.094208 [05/23/2020-11:13:12] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:13:12] [V] [TRT] Tactic: 5 time 0.186368 [05/23/2020-11:13:12] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.044032 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:13:12] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:12] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:13:12] [V] [TRT] [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:13:12] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:12] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:13:12] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:12] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:13:12] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:13:12] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: 1825138533642645384 time 0.262144 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: 3915320020053085238 time 0.260096 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: 6808617066150061604 time 0.152576 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: -8060443123034038864 time 0.162816 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: -4420849921117327522 time 0.145408 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: -3946921629105938337 time 0.183296 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.145408 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.09728 [05/23/2020-11:13:12] [V] [TRT] Tactic: 1 time 0.15872 [05/23/2020-11:13:12] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:13:12] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:13:12] [V] [TRT] Tactic: 5 time 0.3544 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.09728 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:13:12] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:12] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:13:12] [V] [TRT] [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:12] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:12] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:12] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:12] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] Formats and tactics selection completed in 0.6188 seconds. [05/23/2020-11:13:12] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:13:12] [V] [TRT] Block size 1073741824 [05/23/2020-11:13:12] [V] [TRT] Block size 153600 [05/23/2020-11:13:12] [V] [TRT] Block size 153600 [05/23/2020-11:13:12] [V] [TRT] Block size 2048 [05/23/2020-11:13:12] [V] [TRT] Block size 2048 [05/23/2020-11:13:12] [V] [TRT] Block size 2048 [05/23/2020-11:13:12] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:13:12] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:13:12] [V] [TRT] Engine generation completed in 2.70477 seconds. [05/23/2020-11:13:12] [V] [TRT] Engine Layer Information: [05/23/2020-11:13:12] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:13:12] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:12] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:13:12] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:13:12] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:13:12] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:13:12] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:13:12] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:13:12] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:13:12] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:13:12] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:13:12] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:13:12] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:13:12] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:13:12] [V] [TRT] Original: 48 layers [05/23/2020-11:13:12] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:13:12] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:13:12] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:13:12] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:12] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:12] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:12] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:12] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:13:12] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:13:12] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:13:12] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:13:12] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:13:12] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:13:12] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:13:12] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:13:12] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:13:12] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:13:12] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:13:12] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:13:12] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:13:12] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:13:12] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:13:12] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:13:12] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:13:12] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:13:12] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:13:12] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:13:12] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:13:12] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:13:12] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:13:12] [V] [TRT] Graph construction and optimization completed in 0.0185894 seconds. [05/23/2020-11:13:12] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:13:12] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:12] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:13:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:13:12] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:13:12] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:13:12] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:13:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:13:13] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:13:13] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:13:13] [V] [TRT] Tactic: 6808617066150061604 time 0.015392 [05/23/2020-11:13:13] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:13:13] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:13:13] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:13:13] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:13:13] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:13:13] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.01024 [05/23/2020-11:13:13] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:13:13] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:13:13] [V] [TRT] Tactic: 4 time 1.6128 [05/23/2020-11:13:13] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.01024 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:13:13] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:13] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:13:13] [V] [TRT] [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:13] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:13] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:13:13] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:13:13] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:13:13] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:13:13] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:13:13] [V] [TRT] Tactic: -64 time 0.009216 [05/23/2020-11:13:13] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 256 Time: 0.007168 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:13:13] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:13:13] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:13:13] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:13:13] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:13:13] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:13:13] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:13:13] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:13] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:13:13] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:13:13] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:13] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:13] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:13:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:13:14] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:13:14] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:14] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:13:14] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:13:14] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:13:14] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:13:14] [V] [TRT] Formats and tactics selection completed in 1.24937 seconds. [05/23/2020-11:13:14] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:13:14] [V] [TRT] Block size 1073741824 [05/23/2020-11:13:14] [V] [TRT] Block size 38400 [05/23/2020-11:13:14] [V] [TRT] Block size 38400 [05/23/2020-11:13:14] [V] [TRT] Block size 4608 [05/23/2020-11:13:14] [V] [TRT] Block size 2560 [05/23/2020-11:13:14] [V] [TRT] Block size 1024 [05/23/2020-11:13:14] [V] [TRT] Block size 1024 [05/23/2020-11:13:14] [V] [TRT] Block size 0 [05/23/2020-11:13:14] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:13:14] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:13:14] [V] [TRT] Engine generation completed in 1.30045 seconds. [05/23/2020-11:13:14] [V] [TRT] Engine Layer Information: [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:13:14] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:14] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:14] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:14] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:13:14] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:13:14] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:13:14] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:13:14] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:13:14] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:13:14] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:13:14] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:13:14] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:13:14] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:13:14] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:13:14] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:13:14] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:13:14] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:13:14] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:13:14] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:13:14] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:13:14] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:13:14] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:13:14] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:13:14] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:13:14] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:13:14] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:13:14] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:13:14] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:13:14] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:13:14] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:13:14] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:13:14] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:13:14] [V] [TRT] Original: 12 layers [05/23/2020-11:13:14] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:13:14] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:13:14] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:13:14] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:13:14] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:13:14] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:13:14] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:13:14] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:13:14] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:13:14] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:13:14] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:13:14] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:13:14] [V] [TRT] Graph construction and optimization completed in 0.00785124 seconds. [05/23/2020-11:13:14] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:13:14] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:13:14] [V] [TRT] Tactic: 256 time 0.006208 [05/23/2020-11:13:14] [V] [TRT] Tactic: 512 time 0.006176 [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 512 Time: 0.006176 [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:14] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:14] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:13:14] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:13:14] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:13:14] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:13:14] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:14] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:13:14] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:14] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:14] [V] [TRT] Formats and tactics selection completed in 0.0670998 seconds. [05/23/2020-11:13:14] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:13:14] [V] [TRT] Block size 1073741824 [05/23/2020-11:13:14] [V] [TRT] Block size 512 [05/23/2020-11:13:14] [V] [TRT] Block size 512 [05/23/2020-11:13:14] [V] [TRT] Block size 512 [05/23/2020-11:13:14] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:13:14] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:13:14] [V] [TRT] Engine generation completed in 0.352129 seconds. [05/23/2020-11:13:14] [V] [TRT] Engine Layer Information: [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:13:14] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:13:14] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:14] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:14] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 512, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:13:14] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:14] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:14] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:13:14] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:13:14] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread2 load float count:3834 thread0 load float count:3834 thread3 load float count:3834 thread1 load float count:3834 thread4 load float count:3834 thread7 load float count:3834 thread5 load float count:3834 thread6 load float count:3834 thread9 load float count:3834 thread8 load float count:3834 thread10 load float count:3834 thread12 load float count:3834 thread11 load float count:3834 thread13 load float count:3834 thread15 load float count:3834 thread14 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish thread 19 finish thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 1 finish thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 5 finish thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 13 finish The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish thread 10 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:13:26] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:13:26] [V] [TRT] Original: 18 layers [05/23/2020-11:13:26] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:13:26] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:13:26] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:13:26] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:13:26] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:13:26] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:26] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:26] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:26] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:26] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:13:26] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:13:26] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:26] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:26] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:26] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:26] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:13:26] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:13:26] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:13:26] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:13:26] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:13:26] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:13:26] [V] [TRT] Graph construction and optimization completed in 0.00255208 seconds. [05/23/2020-11:13:28] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:28] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:28] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:28] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:28] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:13:28] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:13:28] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:13:28] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:13:28] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:13:28] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:13:28] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:13:28] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:13:28] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:13:28] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:13:28] [V] [TRT] Tactic: -8060443123034038864 time 0.057344 [05/23/2020-11:13:28] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:13:28] [V] [TRT] Tactic: -4420849921117327522 time 0.065568 [05/23/2020-11:13:28] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:13:28] [V] [TRT] Tactic: -3946921629105938337 time 0.077824 [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 time 0.040992 [05/23/2020-11:13:28] [V] [TRT] Tactic: 1 time 0.063488 [05/23/2020-11:13:28] [V] [TRT] Tactic: 2 time 0.08608 [05/23/2020-11:13:28] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:13:28] [V] [TRT] Tactic: 5 time 0.16896 [05/23/2020-11:13:28] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0.040992 [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:13:28] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:28] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:13:28] [V] [TRT] [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:28] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:28] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:13:28] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:28] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:28] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:13:28] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:13:28] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:28] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:13:28] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:13:29] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:13:29] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: 1825138533642645384 time 0.264192 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: 3915320020053085238 time 0.263168 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: 6808617066150061604 time 0.152576 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:13:29] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:13:29] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:13:29] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:13:29] [V] [TRT] Tactic: 5 time 0.360448 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:13:29] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:29] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:13:29] [V] [TRT] [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.007232 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.007232 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:29] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:29] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:29] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:29] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.007264 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.007264 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:29] [V] [TRT] Formats and tactics selection completed in 0.607567 seconds. [05/23/2020-11:13:29] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:13:29] [V] [TRT] Block size 1073741824 [05/23/2020-11:13:29] [V] [TRT] Block size 153600 [05/23/2020-11:13:29] [V] [TRT] Block size 153600 [05/23/2020-11:13:29] [V] [TRT] Block size 2048 [05/23/2020-11:13:29] [V] [TRT] Block size 2048 [05/23/2020-11:13:29] [V] [TRT] Block size 2048 [05/23/2020-11:13:29] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:13:29] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:13:29] [V] [TRT] Engine generation completed in 2.66058 seconds. [05/23/2020-11:13:29] [V] [TRT] Engine Layer Information: [05/23/2020-11:13:29] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:13:29] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:29] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:13:29] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:13:29] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:13:29] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:13:29] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:13:29] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:13:29] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:13:29] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:13:29] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:13:29] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:13:29] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:13:29] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:13:29] [V] [TRT] Original: 48 layers [05/23/2020-11:13:29] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:13:29] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:13:29] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:13:29] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:29] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:29] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:29] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:29] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:13:29] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:13:29] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:13:29] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:13:29] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:13:29] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:13:29] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:13:29] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:13:29] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:13:29] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:13:29] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:13:29] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:13:29] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:13:29] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:13:29] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:13:29] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:13:29] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:13:29] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:13:29] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:13:29] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:13:29] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:13:29] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:13:29] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:13:29] [V] [TRT] Graph construction and optimization completed in 0.0221618 seconds. [05/23/2020-11:13:29] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:13:29] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:29] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:29] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:13:29] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:13:29] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: 1825138533642645384 time 0.018528 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:13:29] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:13:29] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:13:29] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:13:29] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:13:29] [V] [TRT] Tactic: 4 time 1.61997 [05/23/2020-11:13:29] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:13:29] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:29] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:13:29] [V] [TRT] [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:29] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:29] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:29] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 time 0.005184 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0.005184 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:30] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:30] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:13:30] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:13:30] [V] [TRT] Tactic: 256 time 0.007232 [05/23/2020-11:13:30] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:13:30] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:13:30] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:13:30] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 256 Time: 0.007232 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:13:30] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:13:30] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:13:30] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:13:30] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:13:30] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:13:30] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:13:30] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:13:30] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:13:30] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:13:30] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:13:30] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:13:30] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:30] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:30] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:13:30] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:13:30] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:13:30] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:13:30] [V] [TRT] Tactic: 512 time 0.006176 [05/23/2020-11:13:30] [V] [TRT] Fastest Tactic: 512 Time: 0.006176 [05/23/2020-11:13:30] [V] [TRT] Formats and tactics selection completed in 1.30486 seconds. [05/23/2020-11:13:30] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:13:30] [V] [TRT] Block size 1073741824 [05/23/2020-11:13:30] [V] [TRT] Block size 38400 [05/23/2020-11:13:30] [V] [TRT] Block size 38400 [05/23/2020-11:13:30] [V] [TRT] Block size 4608 [05/23/2020-11:13:30] [V] [TRT] Block size 2560 [05/23/2020-11:13:30] [V] [TRT] Block size 1024 [05/23/2020-11:13:30] [V] [TRT] Block size 1024 [05/23/2020-11:13:30] [V] [TRT] Block size 0 [05/23/2020-11:13:30] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:13:30] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:13:31] [V] [TRT] Engine generation completed in 1.35387 seconds. [05/23/2020-11:13:31] [V] [TRT] Engine Layer Information: [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:13:31] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:31] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:31] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:31] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:13:31] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:13:31] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:13:31] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:13:31] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:13:31] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:13:31] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:13:31] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:13:31] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:13:31] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:13:31] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:13:31] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:13:31] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:13:31] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:13:31] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 256, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:13:31] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:13:31] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:13:31] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:13:31] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:13:31] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:13:31] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:13:31] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:13:31] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:13:31] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:13:31] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:13:31] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:13:31] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:13:31] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 512, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:13:31] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:13:31] [V] [TRT] Original: 12 layers [05/23/2020-11:13:31] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:13:31] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:13:31] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:13:31] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:13:31] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:13:31] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:13:31] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:13:31] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:13:31] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:13:31] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:13:31] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:13:31] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:13:31] [V] [TRT] Graph construction and optimization completed in 0.0051953 seconds. [05/23/2020-11:13:31] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:13:31] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:13:31] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:13:31] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:31] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:31] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:31] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:31] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:31] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:31] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:31] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:31] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:13:31] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:13:31] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:13:31] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:13:31] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:13:31] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:13:31] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:31] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:31] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:31] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:31] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:31] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:31] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:31] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:31] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:13:31] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:13:31] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:13:31] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:13:31] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:13:31] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:13:31] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:13:31] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:31] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:31] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:13:31] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:31] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:31] [V] [TRT] Formats and tactics selection completed in 0.069403 seconds. [05/23/2020-11:13:31] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:13:31] [V] [TRT] Block size 1073741824 [05/23/2020-11:13:31] [V] [TRT] Block size 512 [05/23/2020-11:13:31] [V] [TRT] Block size 512 [05/23/2020-11:13:31] [V] [TRT] Block size 512 [05/23/2020-11:13:31] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:13:31] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:13:31] [V] [TRT] Engine generation completed in 0.0871068 seconds. [05/23/2020-11:13:31] [V] [TRT] Engine Layer Information: [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:13:31] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:13:31] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:31] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:31] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:13:31] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:31] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:31] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 256, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:13:31] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:13:31] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:31] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread1 load float count:3834 thread3 load float count:3834 thread0 load float count:3834 thread2 load float count:3834 thread4 load float count:3834 thread7 load float count:3834 thread6 load float count:3834 thread5 load float count:3834 thread8 load float count:3834 thread10 load float count:3834 thread9 load float count:3834 thread11 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread14 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 0 finish The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 16 finish thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 9 finish thread The output sequence length is 65410 finish thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 The output sequence length is 654 thread 1 finish thread 11 finish thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish thread 18 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:13:42] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:13:42] [V] [TRT] Original: 18 layers [05/23/2020-11:13:42] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:13:42] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:13:42] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:13:42] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:13:42] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:13:42] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:42] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:42] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:42] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:42] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:13:42] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:13:42] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:42] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:42] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:42] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:42] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:13:42] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:13:42] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:13:42] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:13:42] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:13:42] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:13:42] [V] [TRT] Graph construction and optimization completed in 0.00245931 seconds. [05/23/2020-11:13:44] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:44] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:44] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:44] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:44] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:13:44] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:13:44] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:13:44] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:13:44] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:13:44] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:13:44] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:13:44] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:13:44] [V] [TRT] Tactic: 6808617066150061604 time 0.055296 [05/23/2020-11:13:44] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:13:44] [V] [TRT] Tactic: -8060443123034038864 time 0.057344 [05/23/2020-11:13:44] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:13:44] [V] [TRT] Tactic: -4420849921117327522 time 0.065536 [05/23/2020-11:13:44] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:13:44] [V] [TRT] Tactic: -3946921629105938337 time 0.078848 [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.055296 [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 time 0.04096 [05/23/2020-11:13:44] [V] [TRT] Tactic: 1 time 0.062464 [05/23/2020-11:13:44] [V] [TRT] Tactic: 2 time 0.08704 [05/23/2020-11:13:44] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:13:44] [V] [TRT] Tactic: 5 time 0.172032 [05/23/2020-11:13:44] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0.04096 [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:13:44] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:44] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:13:44] [V] [TRT] [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:44] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:44] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:13:44] [V] [TRT] Tactic: 1 time 0.00624 [05/23/2020-11:13:44] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 1 Time: 0.00624 [05/23/2020-11:13:44] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:13:44] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:44] [V] [TRT] Tactic: 2 time 0.0072 [05/23/2020-11:13:44] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:44] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:13:45] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:13:45] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: 1825138533642645384 time 0.264192 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: 3915320020053085238 time 0.263168 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: 6808617066150061604 time 0.161792 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: -8060443123034038864 time 0.172032 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: -4420849921117327522 time 0.190464 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: -3946921629105938337 time 0.185344 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.161792 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:13:45] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:13:45] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:13:45] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:13:45] [V] [TRT] Tactic: 5 time 0.355328 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:13:45] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:45] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:13:45] [V] [TRT] [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:45] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:45] [V] [TRT] Tactic: 2 time 0.008256 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:45] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:13:45] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] Formats and tactics selection completed in 0.591968 seconds. [05/23/2020-11:13:45] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:13:45] [V] [TRT] Block size 1073741824 [05/23/2020-11:13:45] [V] [TRT] Block size 153600 [05/23/2020-11:13:45] [V] [TRT] Block size 153600 [05/23/2020-11:13:45] [V] [TRT] Block size 2048 [05/23/2020-11:13:45] [V] [TRT] Block size 2048 [05/23/2020-11:13:45] [V] [TRT] Block size 2048 [05/23/2020-11:13:45] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:13:45] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:13:45] [V] [TRT] Engine generation completed in 2.57502 seconds. [05/23/2020-11:13:45] [V] [TRT] Engine Layer Information: [05/23/2020-11:13:45] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:13:45] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:45] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:13:45] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:13:45] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:13:45] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:13:45] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:13:45] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:13:45] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:13:45] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:13:45] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:13:45] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:13:45] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:13:45] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:13:45] [V] [TRT] Original: 48 layers [05/23/2020-11:13:45] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:13:45] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:13:45] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:13:45] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:45] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:45] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:13:45] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:13:45] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:13:45] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:13:45] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:13:45] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:13:45] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:13:45] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:13:45] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:13:45] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:13:45] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:13:45] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:13:45] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:13:45] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:13:45] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:13:45] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:13:45] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:13:45] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:13:45] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:13:45] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:13:45] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:13:45] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:13:45] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:13:45] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:13:45] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:13:45] [V] [TRT] Graph construction and optimization completed in 0.020008 seconds. [05/23/2020-11:13:45] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:13:45] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:13:45] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:13:45] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:13:45] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:13:45] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.01024 [05/23/2020-11:13:45] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:13:45] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:13:45] [V] [TRT] Tactic: 4 time 1.62099 [05/23/2020-11:13:45] [V] [TRT] Tactic: 5 time 0.036864 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.01024 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:13:45] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:13:45] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:13:45] [V] [TRT] [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:45] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:13:45] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.005216 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.005216 [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:45] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:13:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:13:46] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:13:46] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:13:46] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:13:46] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:13:46] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:13:46] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 128 Time: 0.008192 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:13:46] [V] [TRT] Tactic: 1 time 0.009248 [05/23/2020-11:13:46] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:13:46] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:13:46] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:13:46] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:13:46] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:13:46] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:13:46] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:13:46] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:13:46] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:13:46] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:13:46] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:13:46] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] Formats and tactics selection completed in 1.25284 seconds. [05/23/2020-11:13:46] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:13:46] [V] [TRT] Block size 1073741824 [05/23/2020-11:13:46] [V] [TRT] Block size 38400 [05/23/2020-11:13:46] [V] [TRT] Block size 38400 [05/23/2020-11:13:46] [V] [TRT] Block size 4608 [05/23/2020-11:13:46] [V] [TRT] Block size 2560 [05/23/2020-11:13:46] [V] [TRT] Block size 1024 [05/23/2020-11:13:46] [V] [TRT] Block size 1024 [05/23/2020-11:13:46] [V] [TRT] Block size 0 [05/23/2020-11:13:46] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:13:46] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:13:46] [V] [TRT] Engine generation completed in 1.29779 seconds. [05/23/2020-11:13:46] [V] [TRT] Engine Layer Information: [05/23/2020-11:13:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:13:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:13:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:13:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:13:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:13:46] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:46] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:46] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:13:46] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:13:46] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:13:46] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:13:46] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:13:46] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:13:46] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:13:46] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:13:46] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:13:46] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:13:46] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:13:46] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:13:46] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:13:46] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:13:46] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:13:46] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:13:46] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:13:46] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:13:46] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:13:46] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:13:46] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:13:46] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:13:46] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:13:46] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:13:46] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:13:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:13:46] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:13:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:13:46] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:13:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:13:46] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:13:46] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:13:46] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:13:46] [V] [TRT] Original: 12 layers [05/23/2020-11:13:46] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:13:46] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:13:46] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:13:46] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:13:46] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:13:46] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:13:46] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:13:46] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:13:46] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:13:46] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:13:46] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:13:46] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:13:46] [V] [TRT] Graph construction and optimization completed in 0.0071923 seconds. [05/23/2020-11:13:46] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:13:46] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:13:46] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:13:46] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:13:46] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:13:46] [V] [TRT] Tactic: 0 time 0.005216 [05/23/2020-11:13:46] [V] [TRT] Fastest Tactic: 0 Time: 0.005216 [05/23/2020-11:13:46] [V] [TRT] Formats and tactics selection completed in 0.0650459 seconds. [05/23/2020-11:13:46] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:13:46] [V] [TRT] Block size 1073741824 [05/23/2020-11:13:46] [V] [TRT] Block size 512 [05/23/2020-11:13:46] [V] [TRT] Block size 512 [05/23/2020-11:13:46] [V] [TRT] Block size 512 [05/23/2020-11:13:46] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:13:46] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:13:47] [V] [TRT] Engine generation completed in 0.344377 seconds. [05/23/2020-11:13:47] [V] [TRT] Engine Layer Information: [05/23/2020-11:13:47] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:13:47] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:13:47] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:13:47] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:47] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:47] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:13:47] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:47] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:13:47] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:13:47] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:13:47] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:13:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread4 load float count:3834 thread0 load float count:3834 thread1 load float count:3834 thread2 load float count:3834 thread3 load float count:3834 thread6 load float count:3834 thread7 load float count:3834 thread5 load float count:3834 thread8 load float count:3834 thread9 load float count:3834 thread11 load float count:3834 thread10 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread19 load float count:3834 thread18 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 18 finish thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish thread 15 finish thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:14:00] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:00] [V] [TRT] Original: 18 layers [05/23/2020-11:14:00] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:14:00] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:14:00] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:14:00] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:14:00] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:14:00] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:00] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:00] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:00] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:00] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:14:00] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:14:00] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:00] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:00] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:00] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:00] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:14:00] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:14:00] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:14:00] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:14:00] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:14:00] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:14:00] [V] [TRT] Graph construction and optimization completed in 0.00252588 seconds. [05/23/2020-11:14:02] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:02] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:02] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:02] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:02] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:14:02] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:14:02] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:14:02] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:02] [V] [TRT] Tactic: 1825138533642645384 time 0.09216 [05/23/2020-11:14:02] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:02] [V] [TRT] Tactic: 3915320020053085238 time 0.09216 [05/23/2020-11:14:02] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:02] [V] [TRT] Tactic: 6808617066150061604 time 0.058368 [05/23/2020-11:14:02] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:02] [V] [TRT] Tactic: -8060443123034038864 time 0.062464 [05/23/2020-11:14:02] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:02] [V] [TRT] Tactic: -4420849921117327522 time 0.070656 [05/23/2020-11:14:02] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:02] [V] [TRT] Tactic: -3946921629105938337 time 0.086016 [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.058368 [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 time 0.045056 [05/23/2020-11:14:02] [V] [TRT] Tactic: 1 time 0.067584 [05/23/2020-11:14:02] [V] [TRT] Tactic: 2 time 0.094208 [05/23/2020-11:14:02] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:14:02] [V] [TRT] Tactic: 5 time 0.176128 [05/23/2020-11:14:02] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0.045056 [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:14:02] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:02] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:02] [V] [TRT] [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:02] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:02] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:14:02] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:14:02] [V] [TRT] Tactic: 2 time 0.01024 [05/23/2020-11:14:02] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:14:02] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:14:03] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:14:03] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:14:03] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: 1825138533642645384 time 0.263168 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: 6808617066150061604 time 0.15872 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: -8060443123034038864 time 0.172032 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: -4420849921117327522 time 0.192512 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.15872 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.09728 [05/23/2020-11:14:03] [V] [TRT] Tactic: 1 time 0.15872 [05/23/2020-11:14:03] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:14:03] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:14:03] [V] [TRT] Tactic: 5 time 0.354304 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.09728 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:14:03] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:03] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:03] [V] [TRT] [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:03] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:03] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:03] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:03] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] Formats and tactics selection completed in 0.610648 seconds. [05/23/2020-11:14:03] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:14:03] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:03] [V] [TRT] Block size 153600 [05/23/2020-11:14:03] [V] [TRT] Block size 153600 [05/23/2020-11:14:03] [V] [TRT] Block size 2048 [05/23/2020-11:14:03] [V] [TRT] Block size 2048 [05/23/2020-11:14:03] [V] [TRT] Block size 2048 [05/23/2020-11:14:03] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:14:03] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:14:03] [V] [TRT] Engine generation completed in 2.59809 seconds. [05/23/2020-11:14:03] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:03] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:14:03] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:03] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:14:03] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:14:03] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:14:03] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:14:03] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:14:03] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:14:03] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:14:03] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:14:03] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:14:03] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:14:03] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:14:03] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:03] [V] [TRT] Original: 48 layers [05/23/2020-11:14:03] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:14:03] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:14:03] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:14:03] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:03] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:03] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:03] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:03] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:14:03] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:14:03] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:14:03] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:14:03] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:14:03] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:14:03] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:14:03] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:14:03] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:14:03] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:14:03] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:14:03] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:14:03] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:14:03] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:14:03] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:14:03] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:14:03] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:14:03] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:14:03] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:14:03] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:14:03] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:14:03] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:14:03] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:14:03] [V] [TRT] Graph construction and optimization completed in 0.0211396 seconds. [05/23/2020-11:14:03] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:14:03] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:14:03] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:14:03] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:14:03] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:03] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.01024 [05/23/2020-11:14:03] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:14:03] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:14:03] [V] [TRT] Tactic: 4 time 1.61075 [05/23/2020-11:14:03] [V] [TRT] Tactic: 5 time 0.036864 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.01024 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:14:03] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:03] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:03] [V] [TRT] [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:03] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:03] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:03] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:14:03] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:03] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:03] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:14:04] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:14:04] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:14:04] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:14:04] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:14:04] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:14:04] [V] [TRT] Tactic: -128 time 0.0072 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 256 Time: 0.007168 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:14:04] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:14:04] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:14:04] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:14:04] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:14:04] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:14:04] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:14:04] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:04] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:14:04] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:14:04] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:14:04] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:14:04] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Tactic: 512 time 0.006176 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] Formats and tactics selection completed in 1.24182 seconds. [05/23/2020-11:14:04] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:14:04] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:04] [V] [TRT] Block size 38400 [05/23/2020-11:14:04] [V] [TRT] Block size 38400 [05/23/2020-11:14:04] [V] [TRT] Block size 4608 [05/23/2020-11:14:04] [V] [TRT] Block size 2560 [05/23/2020-11:14:04] [V] [TRT] Block size 1024 [05/23/2020-11:14:04] [V] [TRT] Block size 1024 [05/23/2020-11:14:04] [V] [TRT] Block size 0 [05/23/2020-11:14:04] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:14:04] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:14:04] [V] [TRT] Engine generation completed in 1.28812 seconds. [05/23/2020-11:14:04] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:04] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:14:04] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:14:04] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:14:04] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:14:04] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:14:04] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:04] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:04] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:04] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:14:04] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:14:04] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:14:04] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:14:04] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:14:04] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:14:04] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:04] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:14:04] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:14:04] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:14:04] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:14:04] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:14:04] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:14:04] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:14:04] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:14:04] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:14:04] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:14:04] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:14:04] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:14:04] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:04] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:14:04] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:14:04] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:04] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:14:04] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:14:04] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:14:04] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:14:04] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:14:04] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:14:04] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:14:04] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:14:04] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:04] [V] [TRT] Original: 12 layers [05/23/2020-11:14:04] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:14:04] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:14:04] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:14:04] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:14:04] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:14:04] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:14:04] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:14:04] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:14:04] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:14:04] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:14:04] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:14:04] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:14:04] [V] [TRT] Graph construction and optimization completed in 0.00476774 seconds. [05/23/2020-11:14:04] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:14:04] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:14:04] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:14:04] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 512 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:04] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:14:04] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Tactic: 256 time 0.006176 [05/23/2020-11:14:04] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:04] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:14:04] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:04] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:05] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:14:05] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:14:05] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:14:05] [V] [TRT] Formats and tactics selection completed in 0.335896 seconds. [05/23/2020-11:14:05] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:14:05] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:05] [V] [TRT] Block size 512 [05/23/2020-11:14:05] [V] [TRT] Block size 512 [05/23/2020-11:14:05] [V] [TRT] Block size 512 [05/23/2020-11:14:05] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:14:05] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:14:05] [V] [TRT] Engine generation completed in 0.354288 seconds. [05/23/2020-11:14:05] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:05] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:14:05] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:14:05] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:14:05] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:05] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:05] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 512, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:14:05] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:05] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:05] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:14:05] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:14:05] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread4 load float count:3834 thread0 load float count:3834 thread2 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread5 load float count:3834 thread6 load float count:3834 thread7 load float count:3834 thread9 load float count:3834 thread8 load float count:3834 thread11 load float count:3834 thread10 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread18 load float count:3834 thread17 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 10 finish The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 thread 12 finish The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 19 finish The output sequence length is 654 thread 17 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:14:17] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:17] [V] [TRT] Original: 18 layers [05/23/2020-11:14:17] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:14:17] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:14:17] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:14:17] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:14:17] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:14:17] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:17] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:17] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:17] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:17] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:14:17] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:14:17] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:17] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:17] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:17] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:17] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:14:17] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:14:17] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:14:17] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:14:17] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:14:17] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:14:17] [V] [TRT] Graph construction and optimization completed in 0.0026262 seconds. [05/23/2020-11:14:19] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:19] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 time 0.006208 [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006208 [05/23/2020-11:14:19] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:19] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:14:19] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:14:19] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:14:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:19] [V] [TRT] Tactic: 1825138533642645384 time 0.083968 [05/23/2020-11:14:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:19] [V] [TRT] Tactic: 3915320020053085238 time 0.08192 [05/23/2020-11:14:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:19] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:14:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:19] [V] [TRT] Tactic: -8060443123034038864 time 0.057344 [05/23/2020-11:14:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:19] [V] [TRT] Tactic: -4420849921117327522 time 0.065536 [05/23/2020-11:14:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:19] [V] [TRT] Tactic: -3946921629105938337 time 0.078848 [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 time 0.041984 [05/23/2020-11:14:19] [V] [TRT] Tactic: 1 time 0.063488 [05/23/2020-11:14:19] [V] [TRT] Tactic: 2 time 0.08704 [05/23/2020-11:14:19] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:14:19] [V] [TRT] Tactic: 5 time 0.166912 [05/23/2020-11:14:19] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0.041984 [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:14:19] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:19] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:19] [V] [TRT] [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:14:19] [V] [TRT] Tactic: 1 time 0.00624 [05/23/2020-11:14:19] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 1 Time: 0.00624 [05/23/2020-11:14:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:14:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:14:19] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:19] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:19] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:14:20] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:14:20] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: 1825138533642645384 time 0.264192 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: 3915320020053085238 time 0.263168 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: 6808617066150061604 time 0.1536 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: -3946921629105938337 time 0.185344 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:14:20] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:14:20] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:14:20] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:14:20] [V] [TRT] Tactic: 5 time 0.357376 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:14:20] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:20] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:20] [V] [TRT] [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:20] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:20] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:20] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:20] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:20] [V] [TRT] Formats and tactics selection completed in 0.608239 seconds. [05/23/2020-11:14:20] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:14:20] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:20] [V] [TRT] Block size 153600 [05/23/2020-11:14:20] [V] [TRT] Block size 153600 [05/23/2020-11:14:20] [V] [TRT] Block size 2048 [05/23/2020-11:14:20] [V] [TRT] Block size 2048 [05/23/2020-11:14:20] [V] [TRT] Block size 2048 [05/23/2020-11:14:20] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:14:20] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:14:20] [V] [TRT] Engine generation completed in 2.79451 seconds. [05/23/2020-11:14:20] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:14:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:14:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:14:20] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:14:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:14:20] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:14:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:14:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:14:20] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:14:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:14:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:14:20] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:14:20] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:20] [V] [TRT] Original: 48 layers [05/23/2020-11:14:20] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:14:20] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:14:20] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:14:20] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:20] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:20] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:20] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:20] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:14:20] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:14:20] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:14:20] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:14:20] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:14:20] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:14:20] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:14:20] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:14:20] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:14:20] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:14:20] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:14:20] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:14:20] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:14:20] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:14:20] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:14:20] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:14:20] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:14:20] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:14:20] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:14:20] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:14:20] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:14:20] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:14:20] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:14:20] [V] [TRT] Graph construction and optimization completed in 0.0229254 seconds. [05/23/2020-11:14:20] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:14:20] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:20] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:14:20] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:14:20] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: 1825138533642645384 time 0.018496 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:14:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:20] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:14:20] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:14:20] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:14:20] [V] [TRT] Tactic: 4 time 1.61792 [05/23/2020-11:14:20] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:14:20] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:20] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:20] [V] [TRT] [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:21] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:21] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:14:21] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:14:21] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:14:21] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:14:21] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:14:21] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:14:21] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 512 Time: 0.007168 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:14:21] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:14:21] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Tactic: 3 time 0.009216 [05/23/2020-11:14:21] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:14:21] [V] [TRT] Tactic: 128 time 0.006176 [05/23/2020-11:14:21] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:14:21] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:14:21] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:14:21] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:14:21] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:14:21] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:14:21] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:14:21] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:21] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:14:21] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:14:21] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Tactic: 256 time 0.006176 [05/23/2020-11:14:21] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:14:21] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:21] [V] [TRT] Formats and tactics selection completed in 1.21714 seconds. [05/23/2020-11:14:21] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:14:21] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:21] [V] [TRT] Block size 38400 [05/23/2020-11:14:21] [V] [TRT] Block size 38400 [05/23/2020-11:14:21] [V] [TRT] Block size 4608 [05/23/2020-11:14:21] [V] [TRT] Block size 2560 [05/23/2020-11:14:21] [V] [TRT] Block size 1024 [05/23/2020-11:14:21] [V] [TRT] Block size 1024 [05/23/2020-11:14:21] [V] [TRT] Block size 0 [05/23/2020-11:14:21] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:14:21] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:14:22] [V] [TRT] Engine generation completed in 1.26185 seconds. [05/23/2020-11:14:22] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:14:22] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:22] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:22] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:14:22] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:14:22] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:14:22] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:14:22] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:14:22] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:14:22] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:22] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:14:22] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:14:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:14:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:14:22] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 512, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:14:22] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:14:22] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:14:22] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 256, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:14:22] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:14:22] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:14:22] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:14:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:14:22] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:22] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:14:22] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:14:22] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:22] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:14:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:14:22] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:14:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:14:22] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:14:22] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:22] [V] [TRT] Original: 12 layers [05/23/2020-11:14:22] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:14:22] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:14:22] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:14:22] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:14:22] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:14:22] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:14:22] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:14:22] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:14:22] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:14:22] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:14:22] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:14:22] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:14:22] [V] [TRT] Graph construction and optimization completed in 0.00569181 seconds. [05/23/2020-11:14:22] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:14:22] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:14:22] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:14:22] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:22] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:22] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:22] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:22] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:22] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:22] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:22] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:22] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:14:22] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:14:22] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:22] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:22] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:14:22] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:22] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:22] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:22] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:22] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:22] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:22] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:22] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:22] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:22] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:14:22] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:14:22] [V] [TRT] Tactic: 128 time 0.006176 [05/23/2020-11:14:22] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:22] [V] [TRT] Tactic: 512 time 0.006176 [05/23/2020-11:14:22] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:14:22] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:14:22] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:14:22] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:14:22] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:14:22] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:22] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:22] [V] [TRT] Formats and tactics selection completed in 0.32399 seconds. [05/23/2020-11:14:22] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:14:22] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:22] [V] [TRT] Block size 512 [05/23/2020-11:14:22] [V] [TRT] Block size 512 [05/23/2020-11:14:22] [V] [TRT] Block size 512 [05/23/2020-11:14:22] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:14:22] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:14:22] [V] [TRT] Engine generation completed in 0.3429 seconds. [05/23/2020-11:14:22] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:14:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:14:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:22] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:14:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:22] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 256, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:14:22] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:14:22] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread2 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread5 load float count:3834 thread4 load float count:3834 thread7 load float count:3834 thread6 load float count:3834 thread8 load float count:3834 thread11 load float count:3834 thread10 load float count:3834 thread9 load float count:3834 thread15 load float count:3834 thread14 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread17 load float count:3834 thread16 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 13 finish The output sequence length is 654 thread 7 finish thread 9 finish thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish thread 8 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:14:34] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:34] [V] [TRT] Original: 18 layers [05/23/2020-11:14:34] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:14:34] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:14:34] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:14:34] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:14:34] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:14:34] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:34] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:34] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:34] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:34] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:14:34] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:14:34] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:34] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:34] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:34] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:34] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:14:34] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:14:34] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:14:34] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:14:34] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:14:34] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:14:34] [V] [TRT] Graph construction and optimization completed in 0.00231747 seconds. [05/23/2020-11:14:35] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:35] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:35] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:35] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:14:35] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:14:35] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:14:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:35] [V] [TRT] Tactic: 1825138533642645384 time 0.09216 [05/23/2020-11:14:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:35] [V] [TRT] Tactic: 3915320020053085238 time 0.091136 [05/23/2020-11:14:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:35] [V] [TRT] Tactic: 6808617066150061604 time 0.058368 [05/23/2020-11:14:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:35] [V] [TRT] Tactic: -8060443123034038864 time 0.063488 [05/23/2020-11:14:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:35] [V] [TRT] Tactic: -4420849921117327522 time 0.070656 [05/23/2020-11:14:35] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:35] [V] [TRT] Tactic: -3946921629105938337 time 0.084992 [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.058368 [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 time 0.045056 [05/23/2020-11:14:35] [V] [TRT] Tactic: 1 time 0.067616 [05/23/2020-11:14:35] [V] [TRT] Tactic: 2 time 0.094208 [05/23/2020-11:14:35] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:14:35] [V] [TRT] Tactic: 5 time 0.182272 [05/23/2020-11:14:35] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0.045056 [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:14:35] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:35] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:35] [V] [TRT] [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:35] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:14:35] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:35] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:14:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:14:35] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:35] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:35] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:14:36] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:14:36] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: 1825138533642645384 time 0.263168 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: 6808617066150061604 time 0.159744 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: -4420849921117327522 time 0.145408 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.145408 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:14:36] [V] [TRT] Tactic: 1 time 0.159744 [05/23/2020-11:14:36] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:14:36] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:14:36] [V] [TRT] Tactic: 5 time 0.35328 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:14:36] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:36] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:36] [V] [TRT] [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:36] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:36] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:36] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:36] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.0072 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.0072 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:36] [V] [TRT] Formats and tactics selection completed in 0.608661 seconds. [05/23/2020-11:14:36] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:14:36] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:36] [V] [TRT] Block size 153600 [05/23/2020-11:14:36] [V] [TRT] Block size 153600 [05/23/2020-11:14:36] [V] [TRT] Block size 2048 [05/23/2020-11:14:36] [V] [TRT] Block size 2048 [05/23/2020-11:14:36] [V] [TRT] Block size 2048 [05/23/2020-11:14:36] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:14:36] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:14:36] [V] [TRT] Engine generation completed in 2.58864 seconds. [05/23/2020-11:14:36] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:36] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:14:36] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:36] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:14:36] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:14:36] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:14:36] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:14:36] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:14:36] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:14:36] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:14:36] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:14:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:14:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:14:36] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:14:36] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:36] [V] [TRT] Original: 48 layers [05/23/2020-11:14:36] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:14:36] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:14:36] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:14:36] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:36] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:36] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:36] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:36] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:14:36] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:14:36] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:14:36] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:14:36] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:14:36] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:14:36] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:14:36] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:14:36] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:14:36] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:14:36] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:14:36] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:14:36] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:14:36] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:14:36] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:14:36] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:14:36] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:14:36] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:14:36] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:14:36] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:14:36] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:14:36] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:14:36] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:14:36] [V] [TRT] Graph construction and optimization completed in 0.0221558 seconds. [05/23/2020-11:14:36] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:14:36] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:36] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:14:36] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:14:36] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: 3915320020053085238 time 0.0184 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:14:36] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:36] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:14:36] [V] [TRT] Tactic: 1 time 0.017408 [05/23/2020-11:14:36] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:14:36] [V] [TRT] Tactic: 4 time 1.61997 [05/23/2020-11:14:36] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:14:36] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:36] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:36] [V] [TRT] [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:36] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:37] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:37] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:14:37] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:14:37] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:14:37] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:14:37] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:14:37] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:14:37] [V] [TRT] Tactic: -128 time 0.009216 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 128 Time: 0.008192 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:14:37] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:14:37] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:14:37] [V] [TRT] Tactic: 6 time 0.052224 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:14:37] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:14:37] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:14:37] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:14:37] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:14:37] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:14:37] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:14:37] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:14:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:14:37] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:14:38] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:38] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:14:38] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:14:38] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:38] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:14:38] [V] [TRT] Formats and tactics selection completed in 1.29845 seconds. [05/23/2020-11:14:38] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:14:38] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:38] [V] [TRT] Block size 38400 [05/23/2020-11:14:38] [V] [TRT] Block size 38400 [05/23/2020-11:14:38] [V] [TRT] Block size 4608 [05/23/2020-11:14:38] [V] [TRT] Block size 2560 [05/23/2020-11:14:38] [V] [TRT] Block size 1024 [05/23/2020-11:14:38] [V] [TRT] Block size 1024 [05/23/2020-11:14:38] [V] [TRT] Block size 0 [05/23/2020-11:14:38] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:14:38] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:14:38] [V] [TRT] Engine generation completed in 1.34857 seconds. [05/23/2020-11:14:38] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:14:38] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:38] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:38] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:38] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:14:38] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:14:38] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:14:38] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:14:38] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:14:38] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:14:38] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:38] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:14:38] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:14:38] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:14:38] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:14:38] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:14:38] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:14:38] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:14:38] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:14:38] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:14:38] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:14:38] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:14:38] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:14:38] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:38] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:14:38] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:14:38] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:38] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:14:38] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:14:38] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:14:38] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:14:38] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 256, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:14:38] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:38] [V] [TRT] Original: 12 layers [05/23/2020-11:14:38] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:14:38] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:14:38] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:14:38] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:14:38] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:14:38] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:14:38] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:14:38] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:14:38] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:14:38] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:14:38] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:14:38] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:14:38] [V] [TRT] Graph construction and optimization completed in 0.00665747 seconds. [05/23/2020-11:14:38] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:14:38] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:38] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:14:38] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:38] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:14:38] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:38] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:38] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:14:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:38] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:14:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:38] [V] [TRT] Formats and tactics selection completed in 0.0700448 seconds. [05/23/2020-11:14:38] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:14:38] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:38] [V] [TRT] Block size 512 [05/23/2020-11:14:38] [V] [TRT] Block size 512 [05/23/2020-11:14:38] [V] [TRT] Block size 512 [05/23/2020-11:14:38] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:14:38] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:14:38] [V] [TRT] Engine generation completed in 0.0869447 seconds. [05/23/2020-11:14:38] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:14:38] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:14:38] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:38] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:38] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:14:38] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:38] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:38] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:14:38] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:14:38] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:38] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread1 load float count:3834 thread2 load float count:3834 thread5 load float count:3834 thread3 load float count:3834 thread6 load float count:3834 thread4 load float count:3834 thread7 load float count:3834 thread8 load float count:3834 thread10 load float count:3834 thread9 load float count:3834 thread12 load float count:3834 thread11 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 2 finish The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 10 finish The output sequence length is 654 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 1 finish The output sequence length is 654 thread 3 finish thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 18 finish thread 6 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:14:51] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:51] [V] [TRT] Original: 18 layers [05/23/2020-11:14:51] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:14:51] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:14:51] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:14:51] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:14:51] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:14:51] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:51] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:51] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:51] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:51] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:14:51] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:14:51] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:51] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:51] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:51] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:51] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:14:51] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:14:51] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:14:51] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:14:51] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:14:51] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:14:51] [V] [TRT] Graph construction and optimization completed in 0.00261655 seconds. [05/23/2020-11:14:53] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:14:53] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:14:53] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: 1825138533642645384 time 0.082976 [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: -8060443123034038864 time 0.057344 [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: -4420849921117327522 time 0.065536 [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: -3946921629105938337 time 0.078848 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.041984 [05/23/2020-11:14:53] [V] [TRT] Tactic: 1 time 0.063488 [05/23/2020-11:14:53] [V] [TRT] Tactic: 2 time 0.088064 [05/23/2020-11:14:53] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:14:53] [V] [TRT] Tactic: 5 time 0.173056 [05/23/2020-11:14:53] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.041984 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:14:53] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:53] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:53] [V] [TRT] [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.00624 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.00624 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:14:53] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:53] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:14:53] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:53] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:14:53] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:14:53] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: 1825138533642645384 time 0.262144 [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: 3915320020053085238 time 0.26112 [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: 6808617066150061604 time 0.1536 [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: -8060443123034038864 time 0.164864 [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: -4420849921117327522 time 0.14544 [05/23/2020-11:14:53] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:53] [V] [TRT] Tactic: -3946921629105938337 time 0.185376 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.14544 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:14:53] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:14:53] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:14:53] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:14:53] [V] [TRT] Tactic: 5 time 0.356352 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:14:53] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:53] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:53] [V] [TRT] [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:53] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:53] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:53] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:14:53] [V] [TRT] Tactic: 2 time 0.008256 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:14:53] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:14:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:53] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:53] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:53] [V] [TRT] Formats and tactics selection completed in 0.616258 seconds. [05/23/2020-11:14:53] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:14:53] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:53] [V] [TRT] Block size 153600 [05/23/2020-11:14:53] [V] [TRT] Block size 153600 [05/23/2020-11:14:53] [V] [TRT] Block size 2048 [05/23/2020-11:14:53] [V] [TRT] Block size 2048 [05/23/2020-11:14:53] [V] [TRT] Block size 2048 [05/23/2020-11:14:53] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:14:53] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:14:54] [V] [TRT] Engine generation completed in 2.57071 seconds. [05/23/2020-11:14:54] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:54] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:14:54] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:54] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:14:54] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:14:54] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:14:54] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:14:54] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:14:54] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:14:54] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:14:54] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:14:54] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:14:54] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:14:54] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:14:54] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:54] [V] [TRT] Original: 48 layers [05/23/2020-11:14:54] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:14:54] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:14:54] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:14:54] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:54] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:54] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:14:54] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:14:54] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:14:54] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:14:54] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:14:54] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:14:54] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:14:54] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:14:54] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:14:54] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:14:54] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:14:54] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:14:54] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:14:54] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:14:54] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:14:54] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:14:54] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:14:54] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:14:54] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:14:54] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:14:54] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:14:54] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:14:54] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:14:54] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:14:54] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:14:54] [V] [TRT] Graph construction and optimization completed in 0.0217636 seconds. [05/23/2020-11:14:54] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:14:54] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:14:54] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:14:54] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:14:54] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:14:54] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:14:54] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:14:54] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:14:54] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:14:54] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:14:54] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:14:54] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:14:54] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:14:54] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:14:54] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:14:54] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:14:54] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:14:54] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:14:54] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:14:54] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:14:54] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:14:54] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:14:54] [V] [TRT] Tactic: 4 time 1.61882 [05/23/2020-11:14:54] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:14:54] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:14:54] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:14:54] [V] [TRT] [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 time 0.005184 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0.005184 [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:54] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:14:54] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:14:54] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:14:54] [V] [TRT] Tactic: 256 time 0.007232 [05/23/2020-11:14:54] [V] [TRT] Tactic: 512 time 0.008096 [05/23/2020-11:14:54] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:14:54] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:14:54] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 256 Time: 0.007232 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:14:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:14:54] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:14:54] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:14:54] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:14:54] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:14:54] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:14:54] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:14:55] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:14:55] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:14:55] [V] [TRT] Tactic: -128 time 0.007232 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:14:55] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:14:55] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:14:55] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:14:55] [V] [TRT] Tactic: 1 time 0.006176 [05/23/2020-11:14:55] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006176 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 time 0.006112 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006112 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:14:55] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:14:55] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:14:55] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Tactic: 512 time 0.006176 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:55] [V] [TRT] Formats and tactics selection completed in 1.27694 seconds. [05/23/2020-11:14:55] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:14:55] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:55] [V] [TRT] Block size 38400 [05/23/2020-11:14:55] [V] [TRT] Block size 38400 [05/23/2020-11:14:55] [V] [TRT] Block size 4608 [05/23/2020-11:14:55] [V] [TRT] Block size 2560 [05/23/2020-11:14:55] [V] [TRT] Block size 1024 [05/23/2020-11:14:55] [V] [TRT] Block size 1024 [05/23/2020-11:14:55] [V] [TRT] Block size 0 [05/23/2020-11:14:55] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:14:55] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:14:55] [V] [TRT] Engine generation completed in 1.32654 seconds. [05/23/2020-11:14:55] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:14:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:14:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:14:55] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:14:55] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:14:55] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:14:55] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:14:55] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:14:55] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:14:55] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:14:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:14:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:14:55] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:14:55] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:14:55] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:14:55] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:14:55] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:14:55] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:14:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:14:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:14:55] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:55] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:14:55] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:14:55] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:14:55] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:14:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:14:55] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:14:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:14:55] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:14:55] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:14:55] [V] [TRT] Original: 12 layers [05/23/2020-11:14:55] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:14:55] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:14:55] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:14:55] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:14:55] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:14:55] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:14:55] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:14:55] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:14:55] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:14:55] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:14:55] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:14:55] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:14:55] [V] [TRT] Graph construction and optimization completed in 0.00520975 seconds. [05/23/2020-11:14:55] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:14:55] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:14:55] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:14:55] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:14:55] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:14:55] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Tactic: 512 time 0.007104 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:55] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:14:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:14:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:14:55] [V] [TRT] Formats and tactics selection completed in 0.0681957 seconds. [05/23/2020-11:14:55] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:14:55] [V] [TRT] Block size 1073741824 [05/23/2020-11:14:55] [V] [TRT] Block size 512 [05/23/2020-11:14:55] [V] [TRT] Block size 512 [05/23/2020-11:14:55] [V] [TRT] Block size 512 [05/23/2020-11:14:55] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:14:55] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:14:55] [V] [TRT] Engine generation completed in 0.0858193 seconds. [05/23/2020-11:14:55] [V] [TRT] Engine Layer Information: [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:14:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:14:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:55] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:14:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:14:55] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 256, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:14:55] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:14:55] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:14:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread2 load float count:3834 thread4 load float count:3834 thread0 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread5 load float count:3834 thread6 load float count:3834 thread7 load float count:3834 thread8 load float count:3834 thread9 load float count:3834 thread10 load float count:3834 thread11 load float count:3834 thread12 load float count:3834 thread14 load float count:3834 thread13 load float count:3834 thread16 load float count:3834 thread15 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 [05/23/2020-11:14:55] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:14:55] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:14:55] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:14:55] [E] [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [05/23/2020-11:14:55] [F] [05/23/2020-11:[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... 14:55] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:14:55] [E] [05/23/2020-[TRT] FAILED_EXECUTION: std::exception 11:14:55] [E] [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [05/23/2020-11:14:55] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:14:55] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:14:55] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:14:55] [F] [05/23/[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... 2020-11:14:55] [E] [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [05/23/2020-11:14:55] [F] [05/23/2020-11:14:55] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:14:55] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:14:55] [F] [[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:14:55] [E] 05[TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception /23/2020-11:14:55] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:14:55] [E] [TRT] FAILED_EXECUTION: std::exception stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 12 finish thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish The output sequence length is 1836 thread 0 finish The output sequence length is 1836 thread 3 finish The output sequence length is 1836 thread 13 finish The output sequence length is 1836 The output sequence length is 1836 The output sequence length is 1836 thread 10 finish thread 16 finish thread 19 finish The output sequence length is 1836 thread 18 finish The output sequence length is 1836 thread 11 finish The output sequence length is 1836 thread 17 finish The output sequence length is 1836 thread 14 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:15:16] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:15:16] [V] [TRT] Original: 18 layers [05/23/2020-11:15:16] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:15:16] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:15:16] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:15:16] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:15:16] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:15:16] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:16] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:16] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:16] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:16] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:15:16] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:15:16] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:16] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:16] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:16] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:16] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:15:16] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:15:16] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:15:16] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:15:16] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:15:16] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:15:16] [V] [TRT] Graph construction and optimization completed in 0.0026221 seconds. [05/23/2020-11:15:18] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:18] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:18] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:18] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:15:18] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:15:18] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: -8060443123034038864 time 0.058368 [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: -4420849921117327522 time 0.065536 [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: -3946921629105938337 time 0.078848 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.041984 [05/23/2020-11:15:18] [V] [TRT] Tactic: 1 time 0.062464 [05/23/2020-11:15:18] [V] [TRT] Tactic: 2 time 0.088064 [05/23/2020-11:15:18] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:15:18] [V] [TRT] Tactic: 5 time 0.169984 [05/23/2020-11:15:18] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.041984 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:15:18] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:18] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:15:18] [V] [TRT] [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:15:18] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:18] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:15:18] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:18] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.006176 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006176 [05/23/2020-11:15:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:15:18] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:15:18] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: 1825138533642645384 time 0.264192 [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: 6808617066150061604 time 0.1536 [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:15:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:15:18] [V] [TRT] Tactic: -3946921629105938337 time 0.185344 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:15:18] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:15:18] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:15:18] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:15:18] [V] [TRT] Tactic: 5 time 0.358336 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:15:18] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:18] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:15:18] [V] [TRT] [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:15:19] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:19] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:15:19] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:19] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] Formats and tactics selection completed in 0.846015 seconds. [05/23/2020-11:15:19] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:15:19] [V] [TRT] Block size 1073741824 [05/23/2020-11:15:19] [V] [TRT] Block size 153600 [05/23/2020-11:15:19] [V] [TRT] Block size 153600 [05/23/2020-11:15:19] [V] [TRT] Block size 2048 [05/23/2020-11:15:19] [V] [TRT] Block size 2048 [05/23/2020-11:15:19] [V] [TRT] Block size 2048 [05/23/2020-11:15:19] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:15:19] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:15:19] [V] [TRT] Engine generation completed in 2.5458 seconds. [05/23/2020-11:15:19] [V] [TRT] Engine Layer Information: [05/23/2020-11:15:19] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:15:19] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:15:19] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:15:19] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:15:19] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:15:19] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:15:19] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:15:19] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:15:19] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:15:19] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:15:19] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:15:19] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:15:19] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:15:19] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:15:19] [V] [TRT] Original: 48 layers [05/23/2020-11:15:19] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:15:19] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:15:19] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:15:19] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:19] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:19] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:19] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:19] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:15:19] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:15:19] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:15:19] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:15:19] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:15:19] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:15:19] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:15:19] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:15:19] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:15:19] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:15:19] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:15:19] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:15:19] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:15:19] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:15:19] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:15:19] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:15:19] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:15:19] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:15:19] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:15:19] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:15:19] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:15:19] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:15:19] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:15:19] [V] [TRT] Graph construction and optimization completed in 0.0152537 seconds. [05/23/2020-11:15:19] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:15:19] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:15:19] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:15:19] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:15:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:15:19] [V] [TRT] Tactic: 1825138533642645384 time 0.018432 [05/23/2020-11:15:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:15:19] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:15:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:15:19] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:15:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:15:19] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:15:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:15:19] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:15:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:15:19] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:15:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:15:19] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:15:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:15:19] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:15:19] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:15:19] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:15:19] [V] [TRT] Tactic: 4 time 1.62202 [05/23/2020-11:15:19] [V] [TRT] Tactic: 5 time 0.043008 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:15:19] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:19] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:15:19] [V] [TRT] [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:15:19] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:15:19] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:15:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:19] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:15:20] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:15:20] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:15:20] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:15:20] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:15:20] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:15:20] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 256 Time: 0.007168 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:15:20] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:15:20] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:15:20] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:15:20] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:15:20] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:15:20] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:15:20] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:20] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:15:20] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:15:20] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:15:20] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:15:20] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:15:20] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] Formats and tactics selection completed in 1.15959 seconds. [05/23/2020-11:15:20] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:15:20] [V] [TRT] Block size 1073741824 [05/23/2020-11:15:20] [V] [TRT] Block size 38400 [05/23/2020-11:15:20] [V] [TRT] Block size 38400 [05/23/2020-11:15:20] [V] [TRT] Block size 4608 [05/23/2020-11:15:20] [V] [TRT] Block size 2560 [05/23/2020-11:15:20] [V] [TRT] Block size 1024 [05/23/2020-11:15:20] [V] [TRT] Block size 1024 [05/23/2020-11:15:20] [V] [TRT] Block size 0 [05/23/2020-11:15:20] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:15:20] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:15:20] [V] [TRT] Engine generation completed in 1.21235 seconds. [05/23/2020-11:15:20] [V] [TRT] Engine Layer Information: [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:15:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:15:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:15:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:15:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:15:20] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:15:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:15:20] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:15:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:15:20] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:15:20] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:15:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:15:20] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:15:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:15:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:15:20] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:15:20] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:15:20] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:15:20] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:15:20] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:15:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:15:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:15:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:15:20] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:15:20] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:15:20] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:15:20] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:15:20] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:15:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:15:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:15:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:15:20] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:15:20] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:15:20] [V] [TRT] Original: 12 layers [05/23/2020-11:15:20] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:15:20] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:15:20] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:15:20] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:15:20] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:15:20] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:15:20] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:15:20] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:15:20] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:15:20] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:15:20] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:15:20] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:15:20] [V] [TRT] Graph construction and optimization completed in 0.00533681 seconds. [05/23/2020-11:15:20] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:15:20] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:15:20] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:15:20] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:15:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:20] [V] [TRT] Formats and tactics selection completed in 0.335028 seconds. [05/23/2020-11:15:20] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:15:20] [V] [TRT] Block size 1073741824 [05/23/2020-11:15:20] [V] [TRT] Block size 512 [05/23/2020-11:15:20] [V] [TRT] Block size 512 [05/23/2020-11:15:20] [V] [TRT] Block size 512 [05/23/2020-11:15:20] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:15:20] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:15:20] [V] [TRT] Engine generation completed in 0.356172 seconds. [05/23/2020-11:15:20] [V] [TRT] Engine Layer Information: [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:15:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:15:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:15:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:15:20] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:15:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:15:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:15:20] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 256, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:15:20] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:15:20] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:15:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread3 load float count:3834 thread1 load float count:3834 thread0 load float count:3834 thread2 load float count:3834 thread5 load float count:3834 thread6 load float count:3834 thread4 load float count:3834 thread7 load float count:3834 thread9 load float count:3834 thread8 load float count:3834 thread11 load float count:3834 thread10 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread12 load float count:3834 thread15 load float count:3834 thread17 load float count:3834 thread16 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 6 finish thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 13 finish The output sequence length is 654 thread 0 finish thread 14 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:15:36] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:15:36] [V] [TRT] Original: 18 layers [05/23/2020-11:15:36] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:15:36] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:15:36] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:15:36] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:15:36] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:15:36] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:36] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:36] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:36] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:36] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:15:36] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:15:36] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:36] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:36] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:36] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:36] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:15:36] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:15:36] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:15:36] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:15:36] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:15:36] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:15:36] [V] [TRT] Graph construction and optimization completed in 0.00257602 seconds. [05/23/2020-11:15:38] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:15:38] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:15:38] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: 1825138533642645384 time 0.09216 [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: 3915320020053085238 time 0.091136 [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: 6808617066150061604 time 0.058368 [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: -8060443123034038864 time 0.063488 [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: -4420849921117327522 time 0.070656 [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: -3946921629105938337 time 0.084992 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.058368 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.044032 [05/23/2020-11:15:38] [V] [TRT] Tactic: 1 time 0.068608 [05/23/2020-11:15:38] [V] [TRT] Tactic: 2 time 0.095232 [05/23/2020-11:15:38] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:15:38] [V] [TRT] Tactic: 5 time 0.1864 [05/23/2020-11:15:38] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.044032 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:15:38] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:38] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:15:38] [V] [TRT] [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:15:38] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:38] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:15:38] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:38] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:15:38] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:15:38] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: 1825138533642645384 time 0.263168 [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: 6808617066150061604 time 0.160768 [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: -8060443123034038864 time 0.172032 [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: -4420849921117327522 time 0.192512 [05/23/2020-11:15:38] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:15:38] [V] [TRT] Tactic: -3946921629105938337 time 0.211968 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.160768 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:15:38] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:15:38] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:15:38] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:15:38] [V] [TRT] Tactic: 5 time 0.3584 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:15:38] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:38] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:15:38] [V] [TRT] [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.006176 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006176 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:15:38] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:38] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:15:38] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:38] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:15:38] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:15:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:38] [V] [TRT] Formats and tactics selection completed in 0.595532 seconds. [05/23/2020-11:15:38] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:15:38] [V] [TRT] Block size 1073741824 [05/23/2020-11:15:38] [V] [TRT] Block size 153600 [05/23/2020-11:15:38] [V] [TRT] Block size 153600 [05/23/2020-11:15:38] [V] [TRT] Block size 2048 [05/23/2020-11:15:38] [V] [TRT] Block size 2048 [05/23/2020-11:15:38] [V] [TRT] Block size 2048 [05/23/2020-11:15:38] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:15:38] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:15:39] [V] [TRT] Engine generation completed in 2.56818 seconds. [05/23/2020-11:15:39] [V] [TRT] Engine Layer Information: [05/23/2020-11:15:39] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:15:39] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:15:39] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:15:39] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:15:39] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:15:39] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:15:39] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:15:39] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:15:39] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:15:39] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:15:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:15:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:15:39] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:15:39] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:15:39] [V] [TRT] Original: 48 layers [05/23/2020-11:15:39] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:15:39] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:15:39] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:15:39] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:39] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:39] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:39] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:39] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:15:39] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:15:39] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:15:39] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:15:39] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:15:39] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:15:39] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:15:39] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:15:39] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:15:39] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:15:39] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:15:39] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:15:39] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:15:39] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:15:39] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:15:39] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:15:39] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:15:39] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:15:39] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:15:39] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:15:39] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:15:39] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:15:39] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:15:39] [V] [TRT] Graph construction and optimization completed in 0.0223114 seconds. [05/23/2020-11:15:39] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:15:39] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:39] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:15:39] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:15:39] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:15:39] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:15:39] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:15:39] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:15:39] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:15:39] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:15:39] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:15:39] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:15:39] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:15:39] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:15:39] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:15:39] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:15:39] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:15:39] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:15:39] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:15:39] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:15:39] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:15:39] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:15:39] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:15:39] [V] [TRT] Tactic: 4 time 1.61997 [05/23/2020-11:15:39] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:15:39] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:39] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:15:39] [V] [TRT] [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:15:39] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:39] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:15:39] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:39] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:15:39] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:15:39] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:15:39] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:15:39] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:15:39] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:15:39] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:15:39] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:15:39] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:15:39] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:15:39] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:15:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:15:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:39] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:15:40] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:15:40] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:15:40] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:15:40] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:15:40] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:40] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:15:40] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:15:40] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:15:40] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:15:40] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:15:40] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:15:40] [V] [TRT] Formats and tactics selection completed in 1.27768 seconds. [05/23/2020-11:15:40] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:15:40] [V] [TRT] Block size 1073741824 [05/23/2020-11:15:40] [V] [TRT] Block size 38400 [05/23/2020-11:15:40] [V] [TRT] Block size 38400 [05/23/2020-11:15:40] [V] [TRT] Block size 4608 [05/23/2020-11:15:40] [V] [TRT] Block size 2560 [05/23/2020-11:15:40] [V] [TRT] Block size 1024 [05/23/2020-11:15:40] [V] [TRT] Block size 1024 [05/23/2020-11:15:40] [V] [TRT] Block size 0 [05/23/2020-11:15:40] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:15:40] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:15:40] [V] [TRT] Engine generation completed in 1.32407 seconds. [05/23/2020-11:15:40] [V] [TRT] Engine Layer Information: [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:15:40] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:15:40] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:15:40] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:15:40] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:15:40] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:15:40] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:15:40] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:15:40] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:15:40] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:15:40] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:15:40] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:15:40] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:15:40] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:15:40] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:15:40] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:15:40] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:15:40] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:15:40] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 256, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:15:40] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:15:40] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:15:40] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:15:40] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:15:40] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:15:40] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:15:40] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:15:40] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:15:40] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:15:40] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:15:40] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:15:40] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:15:40] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:15:40] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:15:40] [V] [TRT] Original: 12 layers [05/23/2020-11:15:40] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:15:40] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:15:40] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:15:40] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:15:40] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:15:40] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:15:40] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:15:40] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:15:40] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:15:40] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:15:40] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:15:40] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:15:40] [V] [TRT] Graph construction and optimization completed in 0.00715103 seconds. [05/23/2020-11:15:40] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:15:40] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:15:40] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:40] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:15:40] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:15:40] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:15:40] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:40] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:15:40] [V] [TRT] Tactic: 0 time 0.005184 [05/23/2020-11:15:40] [V] [TRT] Fastest Tactic: 0 Time: 0.005184 [05/23/2020-11:15:40] [V] [TRT] Formats and tactics selection completed in 0.0718692 seconds. [05/23/2020-11:15:40] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:15:40] [V] [TRT] Block size 1073741824 [05/23/2020-11:15:40] [V] [TRT] Block size 512 [05/23/2020-11:15:40] [V] [TRT] Block size 512 [05/23/2020-11:15:40] [V] [TRT] Block size 512 [05/23/2020-11:15:40] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:15:40] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:15:40] [V] [TRT] Engine generation completed in 0.0884727 seconds. [05/23/2020-11:15:40] [V] [TRT] Engine Layer Information: [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:15:40] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:15:40] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:15:40] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:15:40] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:15:40] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:15:40] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:15:40] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:15:40] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:15:40] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:15:40] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread2 load float count:3834 thread1 load float count:3834 thread0 load float count:3834 thread3 load float count:3834 thread6 load float count:3834 thread4 load float count:3834 thread5 load float count:3834 thread7 load float count:3834 thread8 load float count:3834 thread9 load float count:3834 thread10 load float count:3834 thread11 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 [05/23/2020-11:15:40] [F] [05/23/2020-[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... 03:15:40[] 05[F] /23/2020-11:15:[05/23[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... /2020-11:15:40] [F] 40] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:15:40] [E] [TRT] FAILED_EXECUTIONFAILED_EXECUTION ./rtSafe/Weight: [05/23/2020-11:15:40] [E] [05/23/2020-11:[TRT] FAILED_EXECUTIONFAILED_EXECUTION ./rtSafe/Weight: std::exception 15:40] [[E] 05[TRT] FAILED_EXECUTIONFAILED_EXECUTION ./rtSafe/Weight: std::exception FAILED_EXECUTION: std::exception /23/2020-11:15:40] [E] [TRT] FAILED_EXECUTIONFAILED_EXECUTION ./rtSafe/Weight: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish The output sequence length is 1836 thread 14 finish The output sequence length is 1836 thread 10 finish The output sequence length is 1836 thread 11 finish The output sequence length is 1836 thread 9 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:15:57] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:15:57] [V] [TRT] Original: 18 layers [05/23/2020-11:15:57] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:15:57] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:15:57] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:15:57] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:15:57] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:15:57] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:57] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:57] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:57] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:57] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:15:57] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:15:57] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:57] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:57] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:57] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:57] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:15:57] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:15:57] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:15:57] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:15:57] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:15:57] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:15:57] [V] [TRT] Graph construction and optimization completed in 0.00368461 seconds. [05/23/2020-11:15:59] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:15:59] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:15:59] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: -8060443123034038864 time 0.058368 [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: -4420849921117327522 time 0.06656 [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: -3946921629105938337 time 0.078848 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.04096 [05/23/2020-11:15:59] [V] [TRT] Tactic: 1 time 0.063488 [05/23/2020-11:15:59] [V] [TRT] Tactic: 2 time 0.086016 [05/23/2020-11:15:59] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:15:59] [V] [TRT] Tactic: 5 time 0.16896 [05/23/2020-11:15:59] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.04096 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:15:59] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:59] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:15:59] [V] [TRT] [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.008224 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.008224 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:15:59] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:59] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:15:59] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:59] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:15:59] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:15:59] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: 1825138533642645384 time 0.264192 [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: 6808617066150061604 time 0.1536 [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:15:59] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:15:59] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:15:59] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:15:59] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:15:59] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:15:59] [V] [TRT] Tactic: 5 time 0.357376 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:15:59] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:15:59] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:15:59] [V] [TRT] [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:15:59] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:59] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:15:59] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:15:59] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:59] [V] [TRT] Formats and tactics selection completed in 0.629305 seconds. [05/23/2020-11:15:59] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:15:59] [V] [TRT] Block size 1073741824 [05/23/2020-11:15:59] [V] [TRT] Block size 153600 [05/23/2020-11:15:59] [V] [TRT] Block size 153600 [05/23/2020-11:15:59] [V] [TRT] Block size 2048 [05/23/2020-11:15:59] [V] [TRT] Block size 2048 [05/23/2020-11:15:59] [V] [TRT] Block size 2048 [05/23/2020-11:15:59] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:15:59] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:15:59] [V] [TRT] Engine generation completed in 2.59837 seconds. [05/23/2020-11:15:59] [V] [TRT] Engine Layer Information: [05/23/2020-11:15:59] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:15:59] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:15:59] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:15:59] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:15:59] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:15:59] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:15:59] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:15:59] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:15:59] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:15:59] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:15:59] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:15:59] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:15:59] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:15:59] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:15:59] [V] [TRT] Original: 48 layers [05/23/2020-11:15:59] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:15:59] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:15:59] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:15:59] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:59] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:59] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:15:59] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:15:59] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:15:59] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:15:59] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:15:59] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:15:59] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:15:59] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:15:59] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:15:59] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:15:59] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:15:59] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:15:59] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:15:59] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:15:59] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:15:59] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:15:59] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:15:59] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:15:59] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:15:59] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:15:59] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:15:59] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:15:59] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:15:59] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:15:59] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:15:59] [V] [TRT] Graph construction and optimization completed in 0.0152302 seconds. [05/23/2020-11:15:59] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:15:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:15:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:15:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:15:59] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:16:00] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:16:00] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:16:00] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:16:00] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:16:00] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:16:00] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:16:00] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:16:00] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:16:00] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:16:00] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:16:00] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:16:00] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:16:00] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:16:00] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:16:00] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:16:00] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:16:00] [V] [TRT] Tactic: -4420849921117327522 time 0.013312 [05/23/2020-11:16:00] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:16:00] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.013312 [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:16:00] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:16:00] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:16:00] [V] [TRT] Tactic: 4 time 1.62202 [05/23/2020-11:16:00] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:16:00] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:00] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:16:00] [V] [TRT] [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:16:00] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:16:00] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:16:00] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:16:00] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:16:00] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:16:00] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:16:00] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:16:00] [V] [TRT] Tactic: -128 time 0.009216 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 128 Time: 0.008192 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:16:00] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:16:00] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:16:00] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:16:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:16:00] [V] [TRT] Tactic: 128 time 0.006176 [05/23/2020-11:16:00] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:16:00] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:16:00] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:16:00] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 512 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:16:00] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:16:00] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:16:00] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:16:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:16:00] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:00] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:16:00] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:00] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:16:01] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:16:01] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:16:01] [V] [TRT] Formats and tactics selection completed in 1.29255 seconds. [05/23/2020-11:16:01] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:16:01] [V] [TRT] Block size 1073741824 [05/23/2020-11:16:01] [V] [TRT] Block size 38400 [05/23/2020-11:16:01] [V] [TRT] Block size 38400 [05/23/2020-11:16:01] [V] [TRT] Block size 4608 [05/23/2020-11:16:01] [V] [TRT] Block size 2560 [05/23/2020-11:16:01] [V] [TRT] Block size 1024 [05/23/2020-11:16:01] [V] [TRT] Block size 1024 [05/23/2020-11:16:01] [V] [TRT] Block size 0 [05/23/2020-11:16:01] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:16:01] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:16:01] [V] [TRT] Engine generation completed in 1.34086 seconds. [05/23/2020-11:16:01] [V] [TRT] Engine Layer Information: [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:16:01] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:01] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:01] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:16:01] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:16:01] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:16:01] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:16:01] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:16:01] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:16:01] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:16:01] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:16:01] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:16:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:16:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:16:01] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:16:01] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:16:01] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:16:01] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 512, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:16:01] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:16:01] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:16:01] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:16:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:16:01] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:16:01] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:16:01] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:16:01] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:16:01] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:16:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:16:01] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:16:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:16:01] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:16:01] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:16:01] [V] [TRT] Original: 12 layers [05/23/2020-11:16:01] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:16:01] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:16:01] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:16:01] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:16:01] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:16:01] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:16:01] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:16:01] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:16:01] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:16:01] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:16:01] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:16:01] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:16:01] [V] [TRT] Graph construction and optimization completed in 0.00677045 seconds. [05/23/2020-11:16:01] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:16:01] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:16:01] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:01] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:16:01] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:16:01] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:16:01] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:01] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:16:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:01] [V] [TRT] Formats and tactics selection completed in 0.352081 seconds. [05/23/2020-11:16:01] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:16:01] [V] [TRT] Block size 1073741824 [05/23/2020-11:16:01] [V] [TRT] Block size 512 [05/23/2020-11:16:01] [V] [TRT] Block size 512 [05/23/2020-11:16:01] [V] [TRT] Block size 512 [05/23/2020-11:16:01] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:16:01] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:16:01] [V] [TRT] Engine generation completed in 0.371047 seconds. [05/23/2020-11:16:01] [V] [TRT] Engine Layer Information: [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:16:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:16:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:01] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 256, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:16:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:01] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:16:01] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:16:01] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread2 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread4 load float count:3834 thread7 load float count:3834 thread5 load float count:3834 thread6 load float count:3834 thread8 load float count:3834 thread9 load float count:3834 thread11 load float count:3834 thread10 load float count:3834 thread12 load float count:3834 thread14 load float count:3834 thread13 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread18 load float count:3834 thread17 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 10 finish thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish thread 4 finish thread 6 finish The output sequence length is 654 thread 2 finish thread 8 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:16:16] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:16:16] [V] [TRT] Original: 18 layers [05/23/2020-11:16:16] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:16:16] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:16:16] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:16:16] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:16:16] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:16:16] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:16] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:16] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:16] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:16] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:16:16] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:16:16] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:16] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:16] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:16] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:16] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:16:16] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:16:16] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:16:16] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:16:16] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:16:16] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:16:16] [V] [TRT] Graph construction and optimization completed in 0.00200134 seconds. [05/23/2020-11:16:18] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:16:18] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:16:18] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: 1825138533642645384 time 0.09216 [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: 3915320020053085238 time 0.091136 [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: 6808617066150061604 time 0.058368 [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: -8060443123034038864 time 0.063488 [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: -4420849921117327522 time 0.070656 [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: -3946921629105938337 time 0.085984 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.058368 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.045056 [05/23/2020-11:16:18] [V] [TRT] Tactic: 1 time 0.068608 [05/23/2020-11:16:18] [V] [TRT] Tactic: 2 time 0.094208 [05/23/2020-11:16:18] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:16:18] [V] [TRT] Tactic: 5 time 0.186368 [05/23/2020-11:16:18] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.045056 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:16:18] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:18] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:16:18] [V] [TRT] [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:16:18] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:16:18] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:16:18] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:18] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:16:18] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:16:18] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: 1825138533642645384 time 0.262144 [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: 3915320020053085238 time 0.260096 [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: 6808617066150061604 time 0.15872 [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: -8060443123034038864 time 0.171008 [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: -4420849921117327522 time 0.190464 [05/23/2020-11:16:18] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:16:18] [V] [TRT] Tactic: -3946921629105938337 time 0.218112 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.15872 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.09728 [05/23/2020-11:16:18] [V] [TRT] Tactic: 1 time 0.159744 [05/23/2020-11:16:18] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:16:18] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:16:18] [V] [TRT] Tactic: 5 time 0.35328 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.09728 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:16:18] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:18] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:16:18] [V] [TRT] [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:16:18] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:16:18] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:16:18] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:16:18] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:16:18] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:16:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:18] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:18] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:18] [V] [TRT] Formats and tactics selection completed in 0.575993 seconds. [05/23/2020-11:16:18] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:16:18] [V] [TRT] Block size 1073741824 [05/23/2020-11:16:18] [V] [TRT] Block size 153600 [05/23/2020-11:16:18] [V] [TRT] Block size 153600 [05/23/2020-11:16:18] [V] [TRT] Block size 2048 [05/23/2020-11:16:18] [V] [TRT] Block size 2048 [05/23/2020-11:16:18] [V] [TRT] Block size 2048 [05/23/2020-11:16:18] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:16:18] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:16:19] [V] [TRT] Engine generation completed in 2.67173 seconds. [05/23/2020-11:16:19] [V] [TRT] Engine Layer Information: [05/23/2020-11:16:19] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:16:19] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:19] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:16:19] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:16:19] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:16:19] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:16:19] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:16:19] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:16:19] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:16:19] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:16:19] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:16:19] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:16:19] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:16:19] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:16:19] [V] [TRT] Original: 48 layers [05/23/2020-11:16:19] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:16:19] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:16:19] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:16:19] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:19] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:19] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:19] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:19] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:16:19] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:16:19] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:16:19] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:16:19] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:16:19] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:16:19] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:16:19] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:16:19] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:16:19] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:16:19] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:16:19] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:16:19] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:16:19] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:16:19] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:16:19] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:16:19] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:16:19] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:16:19] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:16:19] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:16:19] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:16:19] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:16:19] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:16:19] [V] [TRT] Graph construction and optimization completed in 0.0200881 seconds. [05/23/2020-11:16:19] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:16:19] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:16:19] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:16:19] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:16:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:16:19] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:16:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:16:19] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:16:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:16:19] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:16:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:16:19] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:16:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:16:19] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:16:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:16:19] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:16:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:16:19] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:16:19] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:16:19] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:16:19] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:16:19] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:16:19] [V] [TRT] Tactic: 4 time 1.61792 [05/23/2020-11:16:19] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:16:19] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:19] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:16:19] [V] [TRT] [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:16:19] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:16:19] [V] [TRT] Tactic: 1 time 0.006176 [05/23/2020-11:16:19] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006176 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006176 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:16:19] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:16:19] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:16:19] [V] [TRT] Tactic: 512 time 0.0072 [05/23/2020-11:16:19] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:16:19] [V] [TRT] Tactic: -64 time 0.008224 [05/23/2020-11:16:19] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 256 Time: 0.007168 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:16:19] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:16:19] [V] [TRT] Tactic: 2 time 0.005184 [05/23/2020-11:16:19] [V] [TRT] Tactic: 3 time 0.009216 [05/23/2020-11:16:19] [V] [TRT] Tactic: 6 time 0.049152 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 2 Time: 0.005184 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:16:19] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:16:19] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:16:19] [V] [TRT] Tactic: -128 time 0.0072 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:16:19] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:16:19] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:16:19] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:16:19] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:19] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:19] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:16:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:16:20] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:16:20] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:20] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:16:20] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:16:20] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:16:20] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:16:20] [V] [TRT] Formats and tactics selection completed in 1.26815 seconds. [05/23/2020-11:16:20] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:16:20] [V] [TRT] Block size 1073741824 [05/23/2020-11:16:20] [V] [TRT] Block size 38400 [05/23/2020-11:16:20] [V] [TRT] Block size 38400 [05/23/2020-11:16:20] [V] [TRT] Block size 4608 [05/23/2020-11:16:20] [V] [TRT] Block size 2560 [05/23/2020-11:16:20] [V] [TRT] Block size 1024 [05/23/2020-11:16:20] [V] [TRT] Block size 1024 [05/23/2020-11:16:20] [V] [TRT] Block size 0 [05/23/2020-11:16:20] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:16:20] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:16:20] [V] [TRT] Engine generation completed in 1.3177 seconds. [05/23/2020-11:16:20] [V] [TRT] Engine Layer Information: [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:16:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:16:20] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:16:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:16:20] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:16:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:16:20] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:16:20] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:16:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:16:20] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:16:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:16:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:16:20] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:16:20] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:16:20] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:16:20] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:16:20] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:16:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:16:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:16:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:16:20] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:16:20] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:16:20] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:16:20] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:16:20] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:16:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:16:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:16:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:16:20] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 256, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:16:20] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:16:20] [V] [TRT] Original: 12 layers [05/23/2020-11:16:20] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:16:20] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:16:20] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:16:20] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:16:20] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:16:20] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:16:20] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:16:20] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:16:20] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:16:20] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:16:20] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:16:20] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:16:20] [V] [TRT] Graph construction and optimization completed in 0.00528001 seconds. [05/23/2020-11:16:20] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:16:20] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:16:20] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:16:20] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:20] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:16:20] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:16:20] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:16:20] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:16:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:20] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:16:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:20] [V] [TRT] Formats and tactics selection completed in 0.0696562 seconds. [05/23/2020-11:16:20] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:16:20] [V] [TRT] Block size 1073741824 [05/23/2020-11:16:20] [V] [TRT] Block size 512 [05/23/2020-11:16:20] [V] [TRT] Block size 512 [05/23/2020-11:16:20] [V] [TRT] Block size 512 [05/23/2020-11:16:20] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:16:20] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:16:20] [V] [TRT] Engine generation completed in 0.3533 seconds. [05/23/2020-11:16:20] [V] [TRT] Engine Layer Information: [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:16:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:16:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:20] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:16:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:20] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:16:20] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:16:20] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:20] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread2 load float count:3834 thread4 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread6 load float count:3834 thread5 load float count:3834 thread9 load float count:3834 thread7 load float count:3834 thread8 load float count:3834 thread10 load float count:3834 thread11 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread16 load float count:3834 thread15 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 4 finish thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 10 finish thread 1 finish thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:16:32] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:16:32] [V] [TRT] Original: 18 layers [05/23/2020-11:16:32] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:16:32] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:16:32] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:16:32] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:16:32] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:16:32] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:32] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:32] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:32] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:32] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:16:32] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:16:32] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:32] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:32] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:32] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:32] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:16:32] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:16:32] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:16:32] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:16:32] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:16:32] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:16:32] [V] [TRT] Graph construction and optimization completed in 0.00286325 seconds. [05/23/2020-11:16:34] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:34] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:34] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:34] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:34] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:16:34] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:16:34] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: 1825138533642645384 time 0.083008 [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: -8060443123034038864 time 0.057376 [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: -4420849921117327522 time 0.06656 [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: -3946921629105938337 time 0.078848 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.041984 [05/23/2020-11:16:34] [V] [TRT] Tactic: 1 time 0.063488 [05/23/2020-11:16:34] [V] [TRT] Tactic: 2 time 0.08704 [05/23/2020-11:16:34] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:16:34] [V] [TRT] Tactic: 5 time 0.171008 [05/23/2020-11:16:34] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.041984 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:16:34] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:34] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:16:34] [V] [TRT] [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:34] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:16:34] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:16:34] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:16:34] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:16:34] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:16:34] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:34] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:16:34] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:16:34] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: 1825138533642645384 time 0.263168 [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: 3915320020053085238 time 0.26112 [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: 6808617066150061604 time 0.152576 [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:16:34] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:16:34] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:16:34] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:16:34] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:16:34] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:16:34] [V] [TRT] Tactic: 5 time 0.356352 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:16:34] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:34] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:16:34] [V] [TRT] [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:34] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:34] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:16:34] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:16:34] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:16:34] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:16:34] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:16:35] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:16:35] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] Formats and tactics selection completed in 0.631807 seconds. [05/23/2020-11:16:35] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:16:35] [V] [TRT] Block size 1073741824 [05/23/2020-11:16:35] [V] [TRT] Block size 153600 [05/23/2020-11:16:35] [V] [TRT] Block size 153600 [05/23/2020-11:16:35] [V] [TRT] Block size 2048 [05/23/2020-11:16:35] [V] [TRT] Block size 2048 [05/23/2020-11:16:35] [V] [TRT] Block size 2048 [05/23/2020-11:16:35] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:16:35] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:16:35] [V] [TRT] Engine generation completed in 2.65095 seconds. [05/23/2020-11:16:35] [V] [TRT] Engine Layer Information: [05/23/2020-11:16:35] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:16:35] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:35] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:16:35] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:16:35] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:16:35] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:16:35] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:16:35] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:16:35] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:16:35] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:16:35] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:16:35] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:16:35] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:16:35] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:16:35] [V] [TRT] Original: 48 layers [05/23/2020-11:16:35] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:16:35] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:16:35] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:16:35] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:35] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:35] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:35] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:35] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:16:35] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:16:35] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:16:35] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:16:35] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:16:35] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:16:35] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:16:35] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:16:35] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:16:35] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:16:35] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:16:35] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:16:35] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:16:35] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:16:35] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:16:35] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:16:35] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:16:35] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:16:35] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:16:35] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:16:35] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:16:35] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:16:35] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:16:35] [V] [TRT] Graph construction and optimization completed in 0.0199915 seconds. [05/23/2020-11:16:35] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:16:35] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:16:35] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:16:35] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:16:35] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:16:35] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:16:35] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:16:35] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:16:35] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:16:35] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:16:35] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:16:35] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:16:35] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:16:35] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:16:35] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:16:35] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:16:35] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:16:35] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:16:35] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:16:35] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:16:35] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:16:35] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:16:35] [V] [TRT] Tactic: 4 time 1.61997 [05/23/2020-11:16:35] [V] [TRT] Tactic: 5 time 0.038912 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:16:35] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:16:35] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:16:35] [V] [TRT] [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.006176 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006176 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:16:35] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Tactic: 2 time 0.0072 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:16:35] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:16:35] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:16:35] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:16:35] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:16:35] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:16:35] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:16:35] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 512 Time: 0.007168 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:16:35] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:16:35] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:16:35] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:16:35] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:16:35] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:16:35] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:35] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:35] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:16:36] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:16:36] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:16:36] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:16:36] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:16:36] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:16:36] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:16:36] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:16:36] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:16:36] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:16:36] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:16:36] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:16:36] [V] [TRT] Formats and tactics selection completed in 1.27144 seconds. [05/23/2020-11:16:36] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:16:36] [V] [TRT] Block size 1073741824 [05/23/2020-11:16:36] [V] [TRT] Block size 38400 [05/23/2020-11:16:36] [V] [TRT] Block size 38400 [05/23/2020-11:16:36] [V] [TRT] Block size 4608 [05/23/2020-11:16:36] [V] [TRT] Block size 2560 [05/23/2020-11:16:36] [V] [TRT] Block size 1024 [05/23/2020-11:16:36] [V] [TRT] Block size 1024 [05/23/2020-11:16:36] [V] [TRT] Block size 0 [05/23/2020-11:16:36] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:16:36] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:16:36] [V] [TRT] Engine generation completed in 1.3195 seconds. [05/23/2020-11:16:36] [V] [TRT] Engine Layer Information: [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:16:36] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:36] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:36] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:16:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:16:36] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:16:36] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:16:36] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:16:36] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:16:36] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:16:36] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:16:36] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:16:36] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:16:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:16:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:16:36] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 512, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:16:36] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:16:36] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:16:36] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:16:36] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:16:36] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:16:36] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:16:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:16:36] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:16:36] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:16:36] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:16:36] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:16:36] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:16:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:16:36] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:16:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:16:36] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:16:36] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:16:36] [V] [TRT] Original: 12 layers [05/23/2020-11:16:36] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:16:36] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:16:36] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:16:36] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:16:36] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:16:36] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:16:36] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:16:36] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:16:36] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:16:36] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:16:36] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:16:36] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:16:36] [V] [TRT] Graph construction and optimization completed in 0.00699908 seconds. [05/23/2020-11:16:36] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:16:36] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:16:36] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:16:36] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:16:36] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:16:36] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:16:36] [V] [TRT] Formats and tactics selection completed in 0.0856705 seconds. [05/23/2020-11:16:36] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:16:36] [V] [TRT] Block size 1073741824 [05/23/2020-11:16:36] [V] [TRT] Block size 512 [05/23/2020-11:16:36] [V] [TRT] Block size 512 [05/23/2020-11:16:36] [V] [TRT] Block size 512 [05/23/2020-11:16:36] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:16:36] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:16:36] [V] [TRT] Engine generation completed in 0.112313 seconds. [05/23/2020-11:16:36] [V] [TRT] Engine Layer Information: [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:16:36] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:16:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:36] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:16:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:36] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:16:36] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:16:36] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:16:36] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:36] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:16:37] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread1 load float count:3834 thread2 load float count:3834 thread0 load float count:3834 thread4 load float count:3834 thread3 load float count:3834 thread7 load float count:3834 thread6 load float count:3834 thread5 load float count:3834 thread9 load float count:3834 thread10 load float count:3834 thread11 load float count:3834 thread8 load float count:3834 thread12 load float count:3834 thread14 load float count:3834 thread13 load float count:3834 thread16 load float count:3834 thread15 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 [05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [E] [TRT] F [05/23/2020-11:16:37] [E] [05/23/2020-11:16:37] [F] [TRT] F [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [F] [05/23[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... /2020-11:16:37] [E] [[TRT] FAILED_EXECUTION: std::exception 05/23/2020-11:16:37] [F] [05/23/2020-11:16:37] [E] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [F] [05/23/2020-[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... 11:16:37] [E] [05/23/2020-11:16:37] [F] [05/23/2020-11:16:37] [TRT] FAILED_EXECUTION: std::exception [E] [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [F] [05/23[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... /2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [F] [05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [F] [05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [E] [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [F] [[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... 05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:16:37] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:16:37] [E] [TRT] FAILED_EXECUTION: std::exception The output sequence length is 1836 thread 5 finish The output sequence length is 1836 thread 2 finish The output sequence length is 1836 thread 0 finish The output sequence length is 1836 thread 16 finish The output sequence length is 1836 thread 8 finish The output sequence length is 1836 thread 7 finish The output sequence length is 1836 thread 14 finish The output sequence length is 1836 thread 18 finish The output sequence length is 1836 thread 12 finish The output sequence length is 1836 thread 13 finish The output sequence length is 1836 The output sequence length is 1836 thread 4 finish thread 3 finish The output sequence length is 1836 thread 11 finish The output sequence length is 1836 thread 17 finish The output sequence length is 1836 thread 6 finish The output sequence length is 1836 thread 19 finish The output sequence length is 1836 thread 15 finish The output sequence length is 1836 thread 10 finish The output sequence length is 1836 The output sequence length is 1836 thread 1 finish thread 9 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:16:58] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:16:58] [V] [TRT] Original: 18 layers [05/23/2020-11:16:58] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:16:58] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:16:58] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:16:58] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:16:58] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:16:58] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:58] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:58] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:58] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:58] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:16:58] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:16:58] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:58] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:58] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:16:58] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:16:58] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:16:58] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:16:58] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:16:58] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:16:58] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:16:58] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:16:58] [V] [TRT] Graph construction and optimization completed in 0.00274183 seconds. [05/23/2020-11:17:00] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:17:00] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:17:00] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: 1825138533642645384 time 0.09216 [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: 3915320020053085238 time 0.091136 [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: 6808617066150061604 time 0.059392 [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: -8060443123034038864 time 0.063488 [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: -4420849921117327522 time 0.070656 [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: -3946921629105938337 time 0.084992 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.059392 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.045056 [05/23/2020-11:17:00] [V] [TRT] Tactic: 1 time 0.068608 [05/23/2020-11:17:00] [V] [TRT] Tactic: 2 time 0.094208 [05/23/2020-11:17:00] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:17:00] [V] [TRT] Tactic: 5 time 0.188416 [05/23/2020-11:17:00] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.045056 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:17:00] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:00] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:00] [V] [TRT] [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:17:00] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:17:00] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:17:00] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:00] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:17:00] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:17:00] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: 1825138533642645384 time 0.262144 [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: 3915320020053085238 time 0.260096 [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: 6808617066150061604 time 0.15872 [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: -8060443123034038864 time 0.170944 [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: -4420849921117327522 time 0.145408 [05/23/2020-11:17:00] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:00] [V] [TRT] Tactic: -3946921629105938337 time 0.183296 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.145408 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.09728 [05/23/2020-11:17:00] [V] [TRT] Tactic: 1 time 0.15872 [05/23/2020-11:17:00] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:17:00] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:17:00] [V] [TRT] Tactic: 5 time 0.3584 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.09728 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:17:00] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:00] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:00] [V] [TRT] [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:00] [V] [TRT] Tactic: 1 time 0.006176 [05/23/2020-11:17:00] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 1 Time: 0.006176 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:00] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:00] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:17:00] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:17:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:00] [V] [TRT] Formats and tactics selection completed in 0.623579 seconds. [05/23/2020-11:17:00] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:17:00] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:00] [V] [TRT] Block size 153600 [05/23/2020-11:17:00] [V] [TRT] Block size 153600 [05/23/2020-11:17:00] [V] [TRT] Block size 2048 [05/23/2020-11:17:00] [V] [TRT] Block size 2048 [05/23/2020-11:17:00] [V] [TRT] Block size 2048 [05/23/2020-11:17:00] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:17:00] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:17:00] [V] [TRT] Engine generation completed in 2.58311 seconds. [05/23/2020-11:17:00] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:00] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:17:00] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:00] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:17:00] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:17:00] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:17:00] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:17:00] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:17:00] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:17:00] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:17:00] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:17:00] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:17:00] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:17:00] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:17:00] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:00] [V] [TRT] Original: 48 layers [05/23/2020-11:17:00] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:17:00] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:17:00] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:17:00] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:00] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:00] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:00] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:00] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:17:00] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:17:00] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:17:00] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:17:00] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:17:00] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:17:00] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:17:00] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:17:00] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:17:00] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:17:00] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:17:00] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:17:00] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:17:00] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:17:00] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:17:00] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:17:00] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:17:00] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:17:00] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:17:00] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:17:00] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:17:00] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:17:00] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:17:00] [V] [TRT] Graph construction and optimization completed in 0.0197455 seconds. [05/23/2020-11:17:01] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:17:01] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:17:01] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:17:01] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:17:01] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:01] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:17:01] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:17:01] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:17:01] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:01] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:17:01] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:17:01] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:17:01] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:01] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:17:01] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:01] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:17:01] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:01] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:17:01] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:01] [V] [TRT] Tactic: -3946921629105938337 time 0.016448 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:17:01] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:17:01] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:17:01] [V] [TRT] Tactic: 4 time 1.61178 [05/23/2020-11:17:01] [V] [TRT] Tactic: 5 time 0.038912 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:17:01] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:01] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:01] [V] [TRT] [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.006208 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006208 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:01] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:01] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:17:01] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:17:01] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:17:01] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:17:01] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:17:01] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:17:01] [V] [TRT] Tactic: -128 time 0.009184 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 128 Time: 0.008192 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:17:01] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:17:01] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:17:01] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:17:01] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 2 Time: 0.007168 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:17:01] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:17:01] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:17:01] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:17:01] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:17:01] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:17:01] [V] [TRT] Tactic: 1 time 0.008128 [05/23/2020-11:17:01] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:17:01] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 1 Time: 0.008128 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:17:01] [V] [TRT] Tactic: 1 time 0.006176 [05/23/2020-11:17:01] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 1 Time: 0.006176 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:17:01] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:01] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:01] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:17:01] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:17:02] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:02] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:02] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:17:02] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:17:02] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:17:02] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:17:02] [V] [TRT] Formats and tactics selection completed in 1.30925 seconds. [05/23/2020-11:17:02] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:17:02] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:02] [V] [TRT] Block size 38400 [05/23/2020-11:17:02] [V] [TRT] Block size 38400 [05/23/2020-11:17:02] [V] [TRT] Block size 4608 [05/23/2020-11:17:02] [V] [TRT] Block size 2560 [05/23/2020-11:17:02] [V] [TRT] Block size 1024 [05/23/2020-11:17:02] [V] [TRT] Block size 1024 [05/23/2020-11:17:02] [V] [TRT] Block size 0 [05/23/2020-11:17:02] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:17:02] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:17:02] [V] [TRT] Engine generation completed in 1.36024 seconds. [05/23/2020-11:17:02] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:17:02] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:02] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:02] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:02] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:17:02] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:17:02] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:17:02] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:17:02] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:17:02] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:17:02] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:02] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:17:02] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:17:02] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:17:02] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:17:02] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:17:02] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:17:02] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:17:02] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:17:02] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:17:02] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:17:02] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:17:02] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:17:02] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:02] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:17:02] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:17:02] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:02] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:17:02] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:17:02] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:17:02] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:17:02] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:17:02] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:02] [V] [TRT] Original: 12 layers [05/23/2020-11:17:02] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:17:02] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:17:02] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:17:02] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:17:02] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:17:02] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:17:02] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:17:02] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:17:02] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:17:02] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:17:02] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:17:02] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:17:02] [V] [TRT] Graph construction and optimization completed in 0.00672563 seconds. [05/23/2020-11:17:02] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:02] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:02] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:17:02] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:17:02] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:17:02] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 128 Time: 0.007168 [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:02] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:02] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:02] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:17:02] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:17:02] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:17:02] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 512 Time: 0.006144 [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:17:02] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:02] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:17:02] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:02] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:02] [V] [TRT] Formats and tactics selection completed in 0.0809739 seconds. [05/23/2020-11:17:02] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:17:02] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:02] [V] [TRT] Block size 512 [05/23/2020-11:17:02] [V] [TRT] Block size 512 [05/23/2020-11:17:02] [V] [TRT] Block size 512 [05/23/2020-11:17:02] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:17:02] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:17:02] [V] [TRT] Engine generation completed in 0.100665 seconds. [05/23/2020-11:17:02] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:17:02] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:17:02] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:02] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:02] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:17:02] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:02] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:02] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 512, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:17:02] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:17:02] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread2 load float count:3834 thread4 load float count:3834 thread6 load float count:3834 thread7 load float count:3834 thread5 load float count:3834 thread9 load float count:3834 thread8 load float count:3834 thread10 load float count:3834 thread11 load float count:3834 thread13 load float count:3834 thread12 load float count:3834 thread14 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 [05/23/2020-11:17:02] [F] [[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... 05/23/2020-11:17:02] [F] [05/23/2020-11[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:17::02] [F] 17[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... :02] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:17:02] [E] [[TRT] FAILED_EXECUTION: std::exception ./rtSafe/Weig 05/23/2020-03:17:02] [E] [05/[TRT] FAILED_EXECUTION: std::exception ./rtSafe/Weig [05/23/2020-11:2317/:022020] -11:17:02] [E] [E] [TRT] FAILED_EXECUTION: *refCount > 0 ../rtSafe/Weigh:std::exception : std::exception [TRT] FAILED_EXECUTION: *refCount > 0 ../rtSafe/Weigh:std::exception : std::exception stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 18 finish The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish The output sequence length is 1836 thread 17 finish The output sequence length is 1836 thread 19 finish The output sequence length is 1836 thread 7 finish The output sequence length is 1836 thread 14 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:17:17] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:17] [V] [TRT] Original: 18 layers [05/23/2020-11:17:17] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:17:17] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:17:17] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:17:17] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:17:17] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:17:17] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:17] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:17] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:17] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:17] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:17:17] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:17:17] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:17] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:17] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:17] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:17] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:17:17] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:17:17] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:17:17] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:17:17] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:17:17] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:17:17] [V] [TRT] Graph construction and optimization completed in 0.00246688 seconds. [05/23/2020-11:17:19] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:17:19] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:17:19] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: -8060443123034038864 time 0.057344 [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: -4420849921117327522 time 0.065536 [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: -3946921629105938337 time 0.07792 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.04096 [05/23/2020-11:17:19] [V] [TRT] Tactic: 1 time 0.063488 [05/23/2020-11:17:19] [V] [TRT] Tactic: 2 time 0.08704 [05/23/2020-11:17:19] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:17:19] [V] [TRT] Tactic: 5 time 0.166944 [05/23/2020-11:17:19] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.04096 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:17:19] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:19] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:19] [V] [TRT] [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.008224 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.008224 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.008224 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.008224 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:17:19] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:19] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:17:19] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:19] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:17:19] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:17:19] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: 1825138533642645384 time 0.263168 [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: 6808617066150061604 time 0.15872 [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: -8060443123034038864 time 0.172032 [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:17:19] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:19] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:17:19] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:17:19] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:17:19] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:17:19] [V] [TRT] Tactic: 5 time 0.354304 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:17:19] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:19] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:19] [V] [TRT] [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:19] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:19] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:19] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:19] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:17:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:17:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:17:19] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] Formats and tactics selection completed in 0.83423 seconds. [05/23/2020-11:17:20] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:17:20] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:20] [V] [TRT] Block size 153600 [05/23/2020-11:17:20] [V] [TRT] Block size 153600 [05/23/2020-11:17:20] [V] [TRT] Block size 2048 [05/23/2020-11:17:20] [V] [TRT] Block size 2048 [05/23/2020-11:17:20] [V] [TRT] Block size 2048 [05/23/2020-11:17:20] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:17:20] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:17:20] [V] [TRT] Engine generation completed in 2.58751 seconds. [05/23/2020-11:17:20] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:17:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:17:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:17:20] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:17:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:17:20] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:17:20] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:17:20] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:17:20] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:17:20] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:17:20] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:17:20] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:17:20] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:20] [V] [TRT] Original: 48 layers [05/23/2020-11:17:20] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:17:20] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:17:20] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:17:20] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:20] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:20] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:20] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:20] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:17:20] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:17:20] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:17:20] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:17:20] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:17:20] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:17:20] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:17:20] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:17:20] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:17:20] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:17:20] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:17:20] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:17:20] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:17:20] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:17:20] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:17:20] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:17:20] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:17:20] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:17:20] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:17:20] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:17:20] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:17:20] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:17:20] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:17:20] [V] [TRT] Graph construction and optimization completed in 0.0206418 seconds. [05/23/2020-11:17:20] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:17:20] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:17:20] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:17:20] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:17:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:20] [V] [TRT] Tactic: 1825138533642645384 time 0.018432 [05/23/2020-11:17:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:17:20] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:17:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:20] [V] [TRT] Tactic: 3915320020053085238 time 0.017408 [05/23/2020-11:17:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:17:20] [V] [TRT] Tactic: 6448355332020552203 time 0.018432 [05/23/2020-11:17:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:20] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:17:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:20] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:17:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:20] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:17:20] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:20] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:17:20] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:17:20] [V] [TRT] Tactic: 2 time 0.017472 [05/23/2020-11:17:20] [V] [TRT] Tactic: 4 time 1.62202 [05/23/2020-11:17:20] [V] [TRT] Tactic: 5 time 0.036864 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:17:20] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:20] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:20] [V] [TRT] [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 time 0.006176 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006176 [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:20] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:20] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:17:20] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:17:20] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:17:20] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:17:20] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:17:20] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:17:20] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 256 Time: 0.007168 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:17:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:17:20] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:17:20] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:17:20] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:17:20] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:17:20] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:17:20] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:17:21] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:17:21] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:17:21] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:17:21] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:17:21] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:17:21] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:17:21] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:17:21] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:17:21] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:17:21] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:17:21] [V] [TRT] Tactic: 256 time 0.006176 [05/23/2020-11:17:21] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 512 Time: 0.006144 [05/23/2020-11:17:21] [V] [TRT] Formats and tactics selection completed in 1.2818 seconds. [05/23/2020-11:17:21] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:17:21] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:21] [V] [TRT] Block size 38400 [05/23/2020-11:17:21] [V] [TRT] Block size 38400 [05/23/2020-11:17:21] [V] [TRT] Block size 4608 [05/23/2020-11:17:21] [V] [TRT] Block size 2560 [05/23/2020-11:17:21] [V] [TRT] Block size 1024 [05/23/2020-11:17:21] [V] [TRT] Block size 1024 [05/23/2020-11:17:21] [V] [TRT] Block size 0 [05/23/2020-11:17:21] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:17:21] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:17:21] [V] [TRT] Engine generation completed in 1.33161 seconds. [05/23/2020-11:17:21] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:17:21] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:21] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:21] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:21] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:17:21] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:17:21] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:17:21] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:17:21] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:17:21] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:17:21] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:21] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:17:21] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:17:21] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:17:21] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:17:21] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:17:21] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:17:21] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:17:21] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 256, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:17:21] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:17:21] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:17:21] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:17:21] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:17:21] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:21] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:17:21] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:17:21] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:21] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:17:21] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:17:21] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:17:21] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:17:21] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 512, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:17:21] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:21] [V] [TRT] Original: 12 layers [05/23/2020-11:17:21] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:17:21] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:17:21] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:17:21] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:17:21] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:17:21] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:17:21] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:17:21] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:17:21] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:17:21] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:17:21] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:17:21] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:17:21] [V] [TRT] Graph construction and optimization completed in 0.0062645 seconds. [05/23/2020-11:17:21] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:17:21] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Tactic: 512 time 0.006176 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:21] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:17:21] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:21] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:17:21] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:17:21] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:17:21] [V] [TRT] Formats and tactics selection completed in 0.0681345 seconds. [05/23/2020-11:17:21] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:17:21] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:21] [V] [TRT] Block size 512 [05/23/2020-11:17:21] [V] [TRT] Block size 512 [05/23/2020-11:17:21] [V] [TRT] Block size 512 [05/23/2020-11:17:21] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:17:21] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:17:21] [V] [TRT] Engine generation completed in 0.0883327 seconds. [05/23/2020-11:17:21] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:17:21] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:17:21] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:21] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:21] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:17:21] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:21] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:21] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:17:21] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:17:21] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:21] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:22] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread2 load float count:3834 thread4 load float count:3834 thread5 load float count:3834 thread6 load float count:3834 thread7 load float count:3834 thread8 load float count:3834 thread9 load float count:3834 thread10 load float count:3834 thread11 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread16 load float count:3834 thread15 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 5 finish thread 8 finish thread 12 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:17:35] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:35] [V] [TRT] Original: 18 layers [05/23/2020-11:17:35] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:17:35] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:17:35] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:17:35] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:17:35] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:17:35] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:35] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:35] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:35] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:35] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:17:35] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:17:35] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:35] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:35] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:35] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:35] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:17:35] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:17:35] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:17:35] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:17:35] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:17:35] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:17:35] [V] [TRT] Graph construction and optimization completed in 0.00260229 seconds. [05/23/2020-11:17:37] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:17:37] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:17:37] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: -8060443123034038864 time 0.057344 [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: -4420849921117327522 time 0.065536 [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: -3946921629105938337 time 0.077824 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.041984 [05/23/2020-11:17:37] [V] [TRT] Tactic: 1 time 0.063488 [05/23/2020-11:17:37] [V] [TRT] Tactic: 2 time 0.08704 [05/23/2020-11:17:37] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:17:37] [V] [TRT] Tactic: 5 time 0.171008 [05/23/2020-11:17:37] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.041984 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:17:37] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:37] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:37] [V] [TRT] [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.008288 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.008288 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:17:37] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:37] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:17:37] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:37] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:17:37] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:17:37] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: 1825138533642645384 time 0.264192 [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: 6808617066150061604 time 0.1536 [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:17:37] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:37] [V] [TRT] Tactic: -3946921629105938337 time 0.184352 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:17:37] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:17:37] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:17:37] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:17:37] [V] [TRT] Tactic: 5 time 0.3584 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:17:37] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:37] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:37] [V] [TRT] [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:37] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:37] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:37] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:37] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:17:37] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:17:37] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:37] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:37] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:37] [V] [TRT] Formats and tactics selection completed in 0.611157 seconds. [05/23/2020-11:17:37] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:17:37] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:37] [V] [TRT] Block size 153600 [05/23/2020-11:17:37] [V] [TRT] Block size 153600 [05/23/2020-11:17:37] [V] [TRT] Block size 2048 [05/23/2020-11:17:37] [V] [TRT] Block size 2048 [05/23/2020-11:17:37] [V] [TRT] Block size 2048 [05/23/2020-11:17:37] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:17:37] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:17:37] [V] [TRT] Engine generation completed in 2.5635 seconds. [05/23/2020-11:17:37] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:37] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:17:37] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:37] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:17:37] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:17:37] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:17:37] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:17:37] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:17:37] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:17:37] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:17:37] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:17:37] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:17:37] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:17:37] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:17:37] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:37] [V] [TRT] Original: 48 layers [05/23/2020-11:17:37] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:17:37] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:17:37] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:17:37] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:37] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:37] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:37] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:37] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:17:37] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:17:37] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:17:37] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:17:37] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:17:37] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:17:37] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:17:37] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:17:37] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:17:37] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:17:37] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:17:37] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:17:37] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:17:37] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:17:37] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:17:37] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:17:37] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:17:37] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:17:37] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:17:37] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:17:37] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:17:37] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:17:37] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:17:37] [V] [TRT] Graph construction and optimization completed in 0.0463619 seconds. [05/23/2020-11:17:38] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:17:38] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:17:38] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:17:38] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:17:38] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:38] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:17:38] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:17:38] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:17:38] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:38] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:17:38] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:17:38] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:17:38] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:38] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:17:38] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:38] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:17:38] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:38] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:17:38] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:38] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:17:38] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:17:38] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:17:38] [V] [TRT] Tactic: 4 time 1.61894 [05/23/2020-11:17:38] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:17:38] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:38] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:38] [V] [TRT] [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:38] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:38] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Tactic: 2 time 0.006176 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:17:38] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:17:38] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:17:38] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:17:38] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:17:38] [V] [TRT] Tactic: -64 time 0.009216 [05/23/2020-11:17:38] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 128 Time: 0.008192 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:17:38] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:17:38] [V] [TRT] Tactic: 2 time 0.006208 [05/23/2020-11:17:38] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:17:38] [V] [TRT] Tactic: 6 time 0.051168 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 2 Time: 0.006208 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:17:38] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:17:38] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:17:38] [V] [TRT] Tactic: -64 time 0.009216 [05/23/2020-11:17:38] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:17:38] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:38] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:17:38] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:17:38] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:17:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:17:38] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:17:39] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:39] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:17:39] [V] [TRT] Tactic: 128 time 0.007104 [05/23/2020-11:17:39] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:17:39] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 512 Time: 0.006144 [05/23/2020-11:17:39] [V] [TRT] Formats and tactics selection completed in 1.2771 seconds. [05/23/2020-11:17:39] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:17:39] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:39] [V] [TRT] Block size 38400 [05/23/2020-11:17:39] [V] [TRT] Block size 38400 [05/23/2020-11:17:39] [V] [TRT] Block size 4608 [05/23/2020-11:17:39] [V] [TRT] Block size 2560 [05/23/2020-11:17:39] [V] [TRT] Block size 1024 [05/23/2020-11:17:39] [V] [TRT] Block size 1024 [05/23/2020-11:17:39] [V] [TRT] Block size 0 [05/23/2020-11:17:39] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:17:39] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:17:39] [V] [TRT] Engine generation completed in 1.34232 seconds. [05/23/2020-11:17:39] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:17:39] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:39] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:39] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:17:39] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:17:39] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:17:39] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:17:39] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:17:39] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:17:39] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:39] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:17:39] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:17:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:17:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:17:39] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:17:39] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:17:39] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:17:39] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 256, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:17:39] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:17:39] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:17:39] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:17:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:17:39] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:39] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:17:39] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:17:39] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:39] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:17:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:17:39] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:17:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:17:39] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 512, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:17:39] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:39] [V] [TRT] Original: 12 layers [05/23/2020-11:17:39] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:17:39] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:17:39] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:17:39] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:17:39] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:17:39] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:17:39] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:17:39] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:17:39] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:17:39] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:17:39] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:17:39] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:17:39] [V] [TRT] Graph construction and optimization completed in 0.00530418 seconds. [05/23/2020-11:17:39] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:17:39] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:17:39] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:17:39] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:39] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:39] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:17:39] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:17:39] [V] [TRT] Tactic: 256 time 0.006208 [05/23/2020-11:17:39] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 256 Time: 0.006208 [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:17:39] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:39] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:17:39] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:39] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:39] [V] [TRT] Formats and tactics selection completed in 0.0693211 seconds. [05/23/2020-11:17:39] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:17:39] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:39] [V] [TRT] Block size 512 [05/23/2020-11:17:39] [V] [TRT] Block size 512 [05/23/2020-11:17:39] [V] [TRT] Block size 512 [05/23/2020-11:17:39] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:17:39] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:17:39] [V] [TRT] Engine generation completed in 0.0880789 seconds. [05/23/2020-11:17:39] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:17:39] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:17:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:39] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 256, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:17:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:39] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:39] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 256, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:17:39] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:17:39] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:39] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread2 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread4 load float count:3834 thread6 load float count:3834 thread7 load float count:3834 thread5 load float count:3834 thread9 load float count:3834 thread8 load float count:3834 thread10 load float count:3834 thread12 load float count:3834 thread11 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread16 load float count:3834 thread15 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 15 finish thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 1 finish thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 12 finish The output sequence length is 654 thread 9 finish thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 6 finish thread 18 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:17:52] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:52] [V] [TRT] Original: 18 layers [05/23/2020-11:17:52] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:17:52] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:17:52] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:17:52] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:17:52] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:17:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:52] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:17:52] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:17:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:52] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:52] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:52] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:17:52] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:17:52] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:17:52] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:17:52] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:17:52] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:17:52] [V] [TRT] Graph construction and optimization completed in 0.0032236 seconds. [05/23/2020-11:17:54] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:17:54] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:17:54] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: 1825138533642645384 time 0.09216 [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: 3915320020053085238 time 0.091136 [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: 6808617066150061604 time 0.058368 [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: -8060443123034038864 time 0.063488 [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: -4420849921117327522 time 0.070656 [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: -3946921629105938337 time 0.084992 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.058368 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.045056 [05/23/2020-11:17:54] [V] [TRT] Tactic: 1 time 0.067584 [05/23/2020-11:17:54] [V] [TRT] Tactic: 2 time 0.094208 [05/23/2020-11:17:54] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:17:54] [V] [TRT] Tactic: 5 time 0.183296 [05/23/2020-11:17:54] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.045056 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:17:54] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:54] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:54] [V] [TRT] [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:17:54] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:54] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:17:54] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:54] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:17:54] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:17:54] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: 1825138533642645384 time 0.262144 [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: 3915320020053085238 time 0.26112 [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: 6808617066150061604 time 0.152576 [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:17:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:54] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:17:54] [V] [TRT] Tactic: 1 time 0.159744 [05/23/2020-11:17:54] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:17:54] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:17:54] [V] [TRT] Tactic: 5 time 0.357376 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:17:54] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:54] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:54] [V] [TRT] [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:54] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:54] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:54] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:17:54] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:17:54] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:17:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:54] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:54] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:54] [V] [TRT] Formats and tactics selection completed in 0.628252 seconds. [05/23/2020-11:17:54] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:17:54] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:54] [V] [TRT] Block size 153600 [05/23/2020-11:17:54] [V] [TRT] Block size 153600 [05/23/2020-11:17:54] [V] [TRT] Block size 2048 [05/23/2020-11:17:54] [V] [TRT] Block size 2048 [05/23/2020-11:17:54] [V] [TRT] Block size 2048 [05/23/2020-11:17:54] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:17:54] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:17:55] [V] [TRT] Engine generation completed in 2.58474 seconds. [05/23/2020-11:17:55] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:17:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:17:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:17:55] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:17:55] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:17:55] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:17:55] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:17:55] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:17:55] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:17:55] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:17:55] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:17:55] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:17:55] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:55] [V] [TRT] Original: 48 layers [05/23/2020-11:17:55] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:17:55] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:17:55] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:17:55] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:55] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:55] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:17:55] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:17:55] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:17:55] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:17:55] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:17:55] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:17:55] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:17:55] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:17:55] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:17:55] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:17:55] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:17:55] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:17:55] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:17:55] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:17:55] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:17:55] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:17:55] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:17:55] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:17:55] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:17:55] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:17:55] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:17:55] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:17:55] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:17:55] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:17:55] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:17:55] [V] [TRT] Graph construction and optimization completed in 0.0218823 seconds. [05/23/2020-11:17:55] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:17:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Tactic: 2 time 0.012288 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:17:55] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:17:55] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:17:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:17:55] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:17:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:17:55] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:17:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:17:55] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:17:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:17:55] [V] [TRT] Tactic: 6448355332020552203 time 0.018432 [05/23/2020-11:17:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:17:55] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:17:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:17:55] [V] [TRT] Tactic: -8060443123034038864 time 0.016384 [05/23/2020-11:17:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:17:55] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:17:55] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:17:55] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:17:55] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:17:55] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:17:55] [V] [TRT] Tactic: 4 time 1.62198 [05/23/2020-11:17:55] [V] [TRT] Tactic: 5 time 0.036864 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:17:55] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:17:55] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:17:55] [V] [TRT] [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:17:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:17:55] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:17:55] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:17:55] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:17:55] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:17:55] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:17:55] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 128 Time: 0.008192 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:17:55] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:17:55] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:17:55] [V] [TRT] Tactic: 6 time 0.052224 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:17:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:17:55] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:17:55] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:17:55] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:17:55] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:17:55] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:17:55] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:17:55] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:17:55] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:17:55] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:17:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:17:56] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:17:56] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Tactic: 2 time 0.00624 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:17:56] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:17:56] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:17:56] [V] [TRT] Formats and tactics selection completed in 1.25049 seconds. [05/23/2020-11:17:56] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:17:56] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:56] [V] [TRT] Block size 38400 [05/23/2020-11:17:56] [V] [TRT] Block size 38400 [05/23/2020-11:17:56] [V] [TRT] Block size 4608 [05/23/2020-11:17:56] [V] [TRT] Block size 2560 [05/23/2020-11:17:56] [V] [TRT] Block size 1024 [05/23/2020-11:17:56] [V] [TRT] Block size 1024 [05/23/2020-11:17:56] [V] [TRT] Block size 0 [05/23/2020-11:17:56] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:17:56] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:17:56] [V] [TRT] Engine generation completed in 1.29541 seconds. [05/23/2020-11:17:56] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:17:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:17:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:17:56] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:17:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:17:56] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:17:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:17:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:17:56] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:17:56] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:17:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:17:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:17:56] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:17:56] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:17:56] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:17:56] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:17:56] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:17:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:17:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:17:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:17:56] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:17:56] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:17:56] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:17:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:17:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:17:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:17:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:17:56] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 256, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:17:56] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:17:56] [V] [TRT] Original: 12 layers [05/23/2020-11:17:56] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:17:56] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:17:56] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:17:56] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:17:56] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:17:56] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:17:56] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:17:56] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:17:56] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:17:56] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:17:56] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:17:56] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:17:56] [V] [TRT] Graph construction and optimization completed in 0.00484855 seconds. [05/23/2020-11:17:56] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:17:56] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:17:56] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:17:56] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Tactic: 256 time 0.006176 [05/23/2020-11:17:56] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:56] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:17:56] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:17:56] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:17:56] [V] [TRT] Formats and tactics selection completed in 0.0652394 seconds. [05/23/2020-11:17:56] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:17:56] [V] [TRT] Block size 1073741824 [05/23/2020-11:17:56] [V] [TRT] Block size 512 [05/23/2020-11:17:56] [V] [TRT] Block size 512 [05/23/2020-11:17:56] [V] [TRT] Block size 512 [05/23/2020-11:17:56] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:17:56] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:17:56] [V] [TRT] Engine generation completed in 0.365547 seconds. [05/23/2020-11:17:56] [V] [TRT] Engine Layer Information: [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:17:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:17:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:56] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:17:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:17:56] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:17:56] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:17:56] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:17:57] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread3 load float count:3834 thread1 load float count:3834 thread2 load float count:3834 thread4 load float count:3834 thread6 load float count:3834 thread5 load float count:3834 thread7 load float count:3834 thread9 load float count:3834 thread8 load float count:3834 thread11 load float count:3834 thread10 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread15 load float count:3834 thread14 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 8 finish thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 14 finish The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 thread 4 finish The output sequence length is 654 thread 10 finish thread 7 finish thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:18:09] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:18:09] [V] [TRT] Original: 18 layers [05/23/2020-11:18:09] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:18:09] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:18:09] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:18:09] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:18:09] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:18:09] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:09] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:09] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:09] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:09] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:18:09] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:18:09] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:09] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:09] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:09] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:09] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:18:09] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:18:09] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:18:09] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:18:09] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:18:09] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:18:09] [V] [TRT] Graph construction and optimization completed in 0.002593 seconds. [05/23/2020-11:18:11] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:11] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:11] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:11] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:11] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:18:11] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:18:11] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: -8060443123034038864 time 0.058368 [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: -4420849921117327522 time 0.06656 [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: -3946921629105938337 time 0.078848 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.041024 [05/23/2020-11:18:11] [V] [TRT] Tactic: 1 time 0.063488 [05/23/2020-11:18:11] [V] [TRT] Tactic: 2 time 0.08704 [05/23/2020-11:18:11] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:18:11] [V] [TRT] Tactic: 5 time 0.173056 [05/23/2020-11:18:11] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.041024 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:18:11] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:11] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:18:11] [V] [TRT] [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.008224 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.008224 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:11] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:18:11] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:11] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:11] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:18:11] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:11] [V] [TRT] Tactic: 2 time 0.008224 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:11] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:18:11] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:18:11] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: 1825138533642645384 time 0.262144 [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: 3915320020053085238 time 0.26112 [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: 6808617066150061604 time 0.1536 [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: -8060443123034038864 time 0.163808 [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:18:11] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:18:11] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.097344 [05/23/2020-11:18:11] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:18:11] [V] [TRT] Tactic: 2 time 0.110592 [05/23/2020-11:18:11] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:18:11] [V] [TRT] Tactic: 5 time 0.357376 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.097344 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:18:11] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:11] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:18:11] [V] [TRT] [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:11] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:11] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:11] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:11] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:12] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:12] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:12] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:12] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.007264 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.007264 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] Formats and tactics selection completed in 0.621968 seconds. [05/23/2020-11:18:12] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:18:12] [V] [TRT] Block size 1073741824 [05/23/2020-11:18:12] [V] [TRT] Block size 153600 [05/23/2020-11:18:12] [V] [TRT] Block size 153600 [05/23/2020-11:18:12] [V] [TRT] Block size 2048 [05/23/2020-11:18:12] [V] [TRT] Block size 2048 [05/23/2020-11:18:12] [V] [TRT] Block size 2048 [05/23/2020-11:18:12] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:18:12] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:18:12] [V] [TRT] Engine generation completed in 2.6329 seconds. [05/23/2020-11:18:12] [V] [TRT] Engine Layer Information: [05/23/2020-11:18:12] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:18:12] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:12] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:18:12] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:18:12] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:18:12] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:18:12] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:18:12] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:18:12] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:18:12] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:18:12] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:18:12] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:18:12] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:18:12] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:18:12] [V] [TRT] Original: 48 layers [05/23/2020-11:18:12] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:18:12] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:18:12] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:18:12] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:12] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:12] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:12] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:12] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:18:12] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:18:12] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:18:12] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:18:12] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:18:12] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:18:12] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:18:12] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:18:12] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:18:12] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:18:12] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:18:12] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:18:12] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:18:12] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:18:12] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:18:12] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:18:12] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:18:12] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:18:12] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:18:12] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:18:12] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:18:12] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:18:12] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:18:12] [V] [TRT] Graph construction and optimization completed in 0.0202213 seconds. [05/23/2020-11:18:12] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:18:12] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:18:12] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:18:12] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:18:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:18:12] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:18:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:18:12] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:18:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:18:12] [V] [TRT] Tactic: 3915320020053085238 time 0.017472 [05/23/2020-11:18:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:18:12] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:18:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:18:12] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:18:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:18:12] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:18:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:18:12] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:18:12] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:18:12] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:18:12] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:18:12] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:18:12] [V] [TRT] Tactic: 4 time 1.61894 [05/23/2020-11:18:12] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:18:12] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:12] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:18:12] [V] [TRT] [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.007072 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.007072 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:12] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:12] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:18:12] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:18:12] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:18:12] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:18:12] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:18:12] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:18:12] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 128 Time: 0.008192 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:18:12] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:18:12] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:18:12] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:18:12] [V] [TRT] Tactic: 6 time 0.052224 [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:18:12] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:18:12] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:12] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:12] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:18:13] [V] [TRT] Tactic: 128 time 0.007136 [05/23/2020-11:18:13] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:18:13] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:18:13] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:18:13] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:18:13] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:18:13] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:18:13] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:18:13] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:18:13] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:18:13] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] Formats and tactics selection completed in 1.27049 seconds. [05/23/2020-11:18:13] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:18:13] [V] [TRT] Block size 1073741824 [05/23/2020-11:18:13] [V] [TRT] Block size 38400 [05/23/2020-11:18:13] [V] [TRT] Block size 38400 [05/23/2020-11:18:13] [V] [TRT] Block size 4608 [05/23/2020-11:18:13] [V] [TRT] Block size 2560 [05/23/2020-11:18:13] [V] [TRT] Block size 1024 [05/23/2020-11:18:13] [V] [TRT] Block size 1024 [05/23/2020-11:18:13] [V] [TRT] Block size 0 [05/23/2020-11:18:13] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:18:13] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:18:13] [V] [TRT] Engine generation completed in 1.31797 seconds. [05/23/2020-11:18:13] [V] [TRT] Engine Layer Information: [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:18:13] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:13] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:13] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:13] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:18:13] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:18:13] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:18:13] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:18:13] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:18:13] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:18:13] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:18:13] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:18:13] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:18:13] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:18:13] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:18:13] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:18:13] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:18:13] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:18:13] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 256, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:18:13] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:18:13] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:18:13] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:18:13] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:18:13] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:18:13] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:18:13] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:18:13] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:18:13] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:18:13] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:18:13] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:18:13] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:18:13] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 256, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:18:13] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:18:13] [V] [TRT] Original: 12 layers [05/23/2020-11:18:13] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:18:13] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:18:13] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:18:13] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:18:13] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:18:13] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:18:13] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:18:13] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:18:13] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:18:13] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:18:13] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:18:13] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:18:13] [V] [TRT] Graph construction and optimization completed in 0.00486154 seconds. [05/23/2020-11:18:13] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:18:13] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:18:13] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:13] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:18:13] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:18:13] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:18:13] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:13] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:13] [V] [TRT] Formats and tactics selection completed in 0.0667482 seconds. [05/23/2020-11:18:13] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:18:13] [V] [TRT] Block size 1073741824 [05/23/2020-11:18:13] [V] [TRT] Block size 512 [05/23/2020-11:18:13] [V] [TRT] Block size 512 [05/23/2020-11:18:13] [V] [TRT] Block size 512 [05/23/2020-11:18:13] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:18:13] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:18:13] [V] [TRT] Engine generation completed in 0.0823819 seconds. [05/23/2020-11:18:13] [V] [TRT] Engine Layer Information: [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:18:13] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:18:13] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:13] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:13] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:18:13] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:13] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:13] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:18:13] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:18:13] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:13] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:14] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread3 load float count:3834 thread2 load float count:3834 thread1 load float count:3834 thread4 load float count:3834 thread5 load float count:3834 thread6 load float count:3834 thread7 load float count:3834 thread8 load float count:3834 thread10 load float count:3834 thread9 load float count:3834 thread11 load float count:3834 thread13 load float count:3834 thread15 load float count:3834 thread14 load float count:3834 thread12 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish thread 7 finish thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 17 finish The output sequence length is 654 thread 14 finish thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:18:22] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:18:22] [V] [TRT] Original: 18 layers [05/23/2020-11:18:22] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:18:22] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:18:22] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:18:22] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:18:22] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:18:22] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:22] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:22] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:22] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:22] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:18:22] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:18:22] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:22] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:22] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:22] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:22] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:18:22] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:18:22] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:18:22] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:18:22] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:18:22] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:18:22] [V] [TRT] Graph construction and optimization completed in 0.00257328 seconds. [05/23/2020-11:18:24] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:24] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:24] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:24] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:24] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:18:24] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:18:24] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:18:24] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:18:24] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:18:24] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:18:24] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:18:24] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:18:24] [V] [TRT] Tactic: 6808617066150061604 time 0.054272 [05/23/2020-11:18:24] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:18:24] [V] [TRT] Tactic: -8060443123034038864 time 0.057344 [05/23/2020-11:18:24] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:18:24] [V] [TRT] Tactic: -4420849921117327522 time 0.06656 [05/23/2020-11:18:24] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:18:24] [V] [TRT] Tactic: -3946921629105938337 time 0.077824 [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.054272 [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 time 0.041024 [05/23/2020-11:18:24] [V] [TRT] Tactic: 1 time 0.062528 [05/23/2020-11:18:24] [V] [TRT] Tactic: 2 time 0.08704 [05/23/2020-11:18:24] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:18:24] [V] [TRT] Tactic: 5 time 0.175104 [05/23/2020-11:18:24] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0.041024 [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:18:24] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:24] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:18:24] [V] [TRT] [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:24] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:24] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:18:24] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:24] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:24] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:18:24] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:18:24] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:24] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:18:24] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:18:25] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:18:25] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: 1825138533642645384 time 0.265216 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: 6808617066150061604 time 0.159744 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:18:25] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:18:25] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:18:25] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:18:25] [V] [TRT] Tactic: 5 time 0.35744 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:18:25] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:25] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:18:25] [V] [TRT] [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:25] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:25] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:25] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:25] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:25] [V] [TRT] Formats and tactics selection completed in 0.605368 seconds. [05/23/2020-11:18:25] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:18:25] [V] [TRT] Block size 1073741824 [05/23/2020-11:18:25] [V] [TRT] Block size 153600 [05/23/2020-11:18:25] [V] [TRT] Block size 153600 [05/23/2020-11:18:25] [V] [TRT] Block size 2048 [05/23/2020-11:18:25] [V] [TRT] Block size 2048 [05/23/2020-11:18:25] [V] [TRT] Block size 2048 [05/23/2020-11:18:25] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:18:25] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:18:25] [V] [TRT] Engine generation completed in 2.53044 seconds. [05/23/2020-11:18:25] [V] [TRT] Engine Layer Information: [05/23/2020-11:18:25] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:18:25] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:25] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:18:25] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:18:25] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:18:25] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:18:25] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:18:25] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:18:25] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:18:25] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:18:25] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:18:25] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:18:25] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:18:25] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:18:25] [V] [TRT] Original: 48 layers [05/23/2020-11:18:25] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:18:25] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:18:25] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:18:25] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:25] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:25] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:25] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:25] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:18:25] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:18:25] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:18:25] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:18:25] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:18:25] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:18:25] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:18:25] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:18:25] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:18:25] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:18:25] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:18:25] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:18:25] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:18:25] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:18:25] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:18:25] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:18:25] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:18:25] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:18:25] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:18:25] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:18:25] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:18:25] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:18:25] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:18:25] [V] [TRT] Graph construction and optimization completed in 0.0222263 seconds. [05/23/2020-11:18:25] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:18:25] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:25] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:18:25] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:18:25] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: 6808617066150061604 time 0.016384 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:18:25] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:18:25] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:18:25] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:18:25] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:18:25] [V] [TRT] Tactic: 4 time 1.62202 [05/23/2020-11:18:25] [V] [TRT] Tactic: 5 time 0.036864 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:18:25] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:25] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:18:25] [V] [TRT] [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.006208 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.006208 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:25] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:25] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:25] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:18:25] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:25] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:18:25] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:26] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Tactic: 2 time 0.00624 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:18:26] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:18:26] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:18:26] [V] [TRT] Tactic: 512 time 0.0072 [05/23/2020-11:18:26] [V] [TRT] Tactic: -32 time 0.00928 [05/23/2020-11:18:26] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:18:26] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 512 Time: 0.0072 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:18:26] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:18:26] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:18:26] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:18:26] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:18:26] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:18:26] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:18:26] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:26] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:18:26] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:18:26] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:18:26] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:18:26] [V] [TRT] Tactic: 128 time 0.006208 [05/23/2020-11:18:26] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Tactic: 512 time 0.00624 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] Formats and tactics selection completed in 1.26553 seconds. [05/23/2020-11:18:26] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:18:26] [V] [TRT] Block size 1073741824 [05/23/2020-11:18:26] [V] [TRT] Block size 38400 [05/23/2020-11:18:26] [V] [TRT] Block size 38400 [05/23/2020-11:18:26] [V] [TRT] Block size 4608 [05/23/2020-11:18:26] [V] [TRT] Block size 2560 [05/23/2020-11:18:26] [V] [TRT] Block size 1024 [05/23/2020-11:18:26] [V] [TRT] Block size 1024 [05/23/2020-11:18:26] [V] [TRT] Block size 0 [05/23/2020-11:18:26] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:18:26] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:18:26] [V] [TRT] Engine generation completed in 1.31272 seconds. [05/23/2020-11:18:26] [V] [TRT] Engine Layer Information: [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:18:26] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:26] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:26] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:26] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:18:26] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:18:26] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:18:26] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:18:26] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:18:26] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:18:26] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:18:26] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:18:26] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:18:26] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:18:26] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:18:26] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 512, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:18:26] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:18:26] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:18:26] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:18:26] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:18:26] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:18:26] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:18:26] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:18:26] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:18:26] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:18:26] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:18:26] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:18:26] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:18:26] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:18:26] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:18:26] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:18:26] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 256, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:18:26] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:18:26] [V] [TRT] Original: 12 layers [05/23/2020-11:18:26] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:18:26] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:18:26] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:18:26] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:18:26] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:18:26] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:18:26] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:18:26] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:18:26] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:18:26] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:18:26] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:18:26] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:18:26] [V] [TRT] Graph construction and optimization completed in 0.00494141 seconds. [05/23/2020-11:18:26] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:18:26] [V] [TRT] Tactic: 128 time 0.006208 [05/23/2020-11:18:26] [V] [TRT] Tactic: 256 time 0.006208 [05/23/2020-11:18:26] [V] [TRT] Tactic: 512 time 0.006176 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 512 Time: 0.006176 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:26] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:18:26] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:18:26] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:18:26] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:26] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:26] [V] [TRT] Formats and tactics selection completed in 0.0680939 seconds. [05/23/2020-11:18:26] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:18:26] [V] [TRT] Block size 1073741824 [05/23/2020-11:18:26] [V] [TRT] Block size 512 [05/23/2020-11:18:26] [V] [TRT] Block size 512 [05/23/2020-11:18:26] [V] [TRT] Block size 512 [05/23/2020-11:18:26] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:18:26] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:18:26] [V] [TRT] Engine generation completed in 0.086129 seconds. [05/23/2020-11:18:26] [V] [TRT] Engine Layer Information: [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:18:26] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:18:26] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:26] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:26] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 512, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:18:26] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:26] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:26] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 256, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:18:26] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:18:26] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:18:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread2 load float count:3834 thread4 load float count:3834 thread3 load float count:3834 thread1 load float count:3834 thread5 load float count:3834 thread7 load float count:3834 thread6 load float count:3834 thread8 load float count:3834 thread9 load float count:3834 thread10 load float count:3834 thread11 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread15 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 10 finish thread 4 finish The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 stop token triggered at step: 327, batch_id: 0, 0.999942 thread 17 finish thread 19 finish The output sequence length is 654 thread 5 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:18:38] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:18:38] [V] [TRT] Original: 18 layers [05/23/2020-11:18:38] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:18:38] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:18:38] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:18:38] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:18:38] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:18:38] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:38] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:38] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:38] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:38] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:18:38] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:18:38] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:38] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:38] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:38] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:38] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:18:38] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:18:38] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:18:38] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:18:38] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:18:38] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:18:38] [V] [TRT] Graph construction and optimization completed in 0.00277441 seconds. [05/23/2020-11:18:40] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:18:40] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:18:40] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: 1825138533642645384 time 0.09216 [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: 3915320020053085238 time 0.091136 [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: 6808617066150061604 time 0.058368 [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: -8060443123034038864 time 0.063488 [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: -4420849921117327522 time 0.070656 [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: -3946921629105938337 time 0.084992 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.058368 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.045056 [05/23/2020-11:18:40] [V] [TRT] Tactic: 1 time 0.068608 [05/23/2020-11:18:40] [V] [TRT] Tactic: 2 time 0.094208 [05/23/2020-11:18:40] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:18:40] [V] [TRT] Tactic: 5 time 0.186368 [05/23/2020-11:18:40] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.045056 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:18:40] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:40] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:18:40] [V] [TRT] [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.009184 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.009184 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:18:40] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:40] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:18:40] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:18:40] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 1 Time: 0.008192 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:18:40] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:18:40] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: 1825138533642645384 time 0.263168 [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: 6808617066150061604 time 0.1536 [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:18:40] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:18:40] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:18:40] [V] [TRT] Tactic: 1 time 0.159744 [05/23/2020-11:18:40] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:18:40] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:18:40] [V] [TRT] Tactic: 5 time 0.360448 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:18:40] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:40] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:18:40] [V] [TRT] [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:40] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:40] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:40] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:40] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:18:40] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:18:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:40] [V] [TRT] Formats and tactics selection completed in 0.630974 seconds. [05/23/2020-11:18:40] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:18:40] [V] [TRT] Block size 1073741824 [05/23/2020-11:18:40] [V] [TRT] Block size 153600 [05/23/2020-11:18:40] [V] [TRT] Block size 153600 [05/23/2020-11:18:40] [V] [TRT] Block size 2048 [05/23/2020-11:18:40] [V] [TRT] Block size 2048 [05/23/2020-11:18:40] [V] [TRT] Block size 2048 [05/23/2020-11:18:40] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:18:40] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:18:41] [V] [TRT] Engine generation completed in 2.6804 seconds. [05/23/2020-11:18:41] [V] [TRT] Engine Layer Information: [05/23/2020-11:18:41] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:18:41] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:41] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:18:41] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:18:41] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:18:41] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:18:41] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:18:41] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:18:41] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:18:41] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:18:41] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:18:41] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:18:41] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:18:41] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:18:41] [V] [TRT] Original: 48 layers [05/23/2020-11:18:41] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:18:41] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:18:41] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:18:41] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:41] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:41] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:18:41] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:18:41] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:18:41] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:18:41] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:18:41] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:18:41] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:18:41] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:18:41] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:18:41] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:18:41] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:18:41] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:18:41] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:18:41] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:18:41] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:18:41] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:18:41] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:18:41] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:18:41] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:18:41] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:18:41] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:18:41] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:18:41] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:18:41] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:18:41] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:18:41] [V] [TRT] Graph construction and optimization completed in 0.0149481 seconds. [05/23/2020-11:18:41] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:18:41] [V] [TRT] Tactic: 1 time 0.005184 [05/23/2020-11:18:41] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 1 Time: 0.005184 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:18:41] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:18:41] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:18:41] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:18:41] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:18:41] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:18:41] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:18:41] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:18:41] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:18:41] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:18:41] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:18:41] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:18:41] [V] [TRT] Tactic: 6808617066150061604 time 0.01536 [05/23/2020-11:18:41] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:18:41] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:18:41] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:18:41] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:18:41] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:18:41] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:18:41] [V] [TRT] Tactic: 1 time 0.017408 [05/23/2020-11:18:41] [V] [TRT] Tactic: 2 time 0.016384 [05/23/2020-11:18:41] [V] [TRT] Tactic: 4 time 1.62099 [05/23/2020-11:18:41] [V] [TRT] Tactic: 5 time 0.037888 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:18:41] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:18:41] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:18:41] [V] [TRT] [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 time 0.005216 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0.005216 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:41] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:18:41] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 time 0.005152 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0.005152 [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:18:41] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:18:41] [V] [TRT] Tactic: 256 time 0.008192 [05/23/2020-11:18:41] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:18:41] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:18:41] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:18:41] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 128 Time: 0.008192 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:18:41] [V] [TRT] Tactic: 1 time 0.008192 [05/23/2020-11:18:41] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Tactic: 3 time 0.010304 [05/23/2020-11:18:41] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:18:41] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:18:41] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:18:41] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:18:41] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:18:41] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:18:41] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:18:41] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:18:41] [V] [TRT] Tactic: 6 time 0.106496 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:18:41] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:41] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:41] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:18:41] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:41] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:41] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:18:42] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:18:42] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:18:42] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:18:42] [V] [TRT] Formats and tactics selection completed in 1.30528 seconds. [05/23/2020-11:18:42] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:18:42] [V] [TRT] Block size 1073741824 [05/23/2020-11:18:42] [V] [TRT] Block size 38400 [05/23/2020-11:18:42] [V] [TRT] Block size 38400 [05/23/2020-11:18:42] [V] [TRT] Block size 4608 [05/23/2020-11:18:42] [V] [TRT] Block size 2560 [05/23/2020-11:18:42] [V] [TRT] Block size 1024 [05/23/2020-11:18:42] [V] [TRT] Block size 1024 [05/23/2020-11:18:42] [V] [TRT] Block size 0 [05/23/2020-11:18:42] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:18:42] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:18:42] [V] [TRT] Engine generation completed in 1.35071 seconds. [05/23/2020-11:18:42] [V] [TRT] Engine Layer Information: [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:18:42] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:42] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:42] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:18:42] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:18:42] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:18:42] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:18:42] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:18:42] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:18:42] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:18:42] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:18:42] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:18:42] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:18:42] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:18:42] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:18:42] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 128, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:18:42] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:18:42] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:18:42] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:18:42] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:18:42] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:18:42] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:18:42] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:18:42] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:18:42] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:18:42] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:18:42] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:18:42] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:18:42] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:18:42] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:18:42] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:18:42] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:18:42] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:18:42] [V] [TRT] Original: 12 layers [05/23/2020-11:18:42] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:18:42] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:18:42] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:18:42] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:18:42] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:18:42] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:18:42] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:18:42] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:18:42] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:18:42] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:18:42] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:18:42] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:18:42] [V] [TRT] Graph construction and optimization completed in 0.00949052 seconds. [05/23/2020-11:18:42] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:18:42] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:18:42] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:18:42] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:18:42] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:42] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:18:42] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:18:42] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:18:42] [V] [TRT] Formats and tactics selection completed in 0.0829555 seconds. [05/23/2020-11:18:42] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:18:42] [V] [TRT] Block size 1073741824 [05/23/2020-11:18:42] [V] [TRT] Block size 512 [05/23/2020-11:18:42] [V] [TRT] Block size 512 [05/23/2020-11:18:42] [V] [TRT] Block size 512 [05/23/2020-11:18:42] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:18:42] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:18:42] [V] [TRT] Engine generation completed in 0.102909 seconds. [05/23/2020-11:18:42] [V] [TRT] Engine Layer Information: [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:18:42] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:18:42] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:42] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:42] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:18:42] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:42] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:18:42] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 256, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:18:42] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:18:42] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:18:42] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread1 load float count:3834 thread0 load float count:3834 thread2 load float count:3834 thread4 load float count:3834 thread3 load float count:3834 thread6 load float count:3834 thread5 load float count:3834 thread7 load float count:3834 thread8 load float count:3834 thread9 load float count:3834 thread10 load float count:3834 thread12 load float count:3834 thread11 load float count:3834 thread13 load float count:3834 thread14 load float count:3834 thread15 load float count:3834 thread17 load float count:3834 thread16 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 [05/23/2020-11:18:42] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] [E] [05/23/2020-11:18:42] [E] [TRT] F [05/23/2020-[TRT] F 11:18:42] [E] [TRT] F [05/23/2020-11:18:42] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:18:42] [F] [05[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] /[F] 23[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... /2020-11[:18:42] 05[F] [[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/052020/23/2020-11:18:42] [F] 0[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23//[202005-/11-232311//[02020-11:0518:2020:-/03:23[18:050/18:421823] /422020-011::42] ] 0[F] 18[:[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... /2020-11:18:42] [F] 05[05/[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... 42:] [E] [05/23/2020-11:18:42] [F] 23/2020-11:18:42] [E] [[TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [F] [05/23/2020-11:18:42] [[F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [E] /[TRT] FAILED_EXECUTION: std::exception [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception 0542/23/] 202005/[0[TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception 23/2020-11:18:42] [E] [05/23/2020-11:18:42] [F] [[E] -11:18:42] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... 23/2020-11:18:42] [E] 05[TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [E] 05[TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:18:42] [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [E] [05/23/2020-11:/1823/2020-11:18::42] /[TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception 42] [E] 23[TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception /[F] 2020-11:18:42] [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [E] [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:18:42] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:18:42] [E] [TRT] FAILED_EXECUTION: std::exception FAILED_EXECUTION: std::exception [05/23/2020-11:18:42] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:18:42] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] [E] [TRT] FAILED_EXECUTION: std::exception [05/23/2020-11:18:42] [F] [TRT] Assertion failed: *refCount > 0 ../rtSafe/WeightsPtr.cpp:20 Aborting... [05/23/2020-11:18:42] [E] [TRT] FAILED_EXECUTION: std::exception The output sequence length is 1836 thread 12 finish The output sequence length is 1836 thread 16 finish The output sequence length is 1836 thread 18 finish The output sequence length is 1836 thread 8 finish The output sequence length is 1836 thread 11 finish The output sequence length is 1836 thread 13 finish The output sequence length is 1836 thread 15 finish The output sequence length is 1836 thread 6 finish The output sequence length is 1836 thread 19 finish The output sequence length is 1836 thread 14 finish The output sequence length is 1836 thread 4 finish The output sequence length is 1836 thread 10 finish The output sequence length is 1836 The output sequence length is 1836 thread 17 finish thread 5 finish The output sequence length is 1836 thread 2 finish The output sequence length is 1836 The output sequence length is 1836 thread 7 finish The output sequence length is 1836 The output sequence length is 1836 thread 3 finish The output sequence length is 1836 thread 1 finish thread 9 finish thread 0 finish tacotron: ../rtSafe/WeightsPtr.cpp:34: void nvinfer1::WeightsPtr::release(): Assertion `*mRefCount > 0' failed. finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:19:37] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:19:37] [V] [TRT] Original: 18 layers [05/23/2020-11:19:37] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:19:37] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:19:37] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:19:37] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:19:37] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:19:37] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:19:37] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:19:37] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:19:37] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:19:37] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:19:37] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:19:37] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:19:37] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:19:37] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:19:37] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:19:37] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:19:37] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:19:37] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:19:37] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:19:37] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:19:37] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:19:37] [V] [TRT] Graph construction and optimization completed in 0.0415621 seconds. [05/23/2020-11:20:16] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:20:16] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:20:16] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: 1825138533642645384 time 0.091136 [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: 3915320020053085238 time 0.091136 [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: 6808617066150061604 time 0.059392 [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: -8060443123034038864 time 0.062464 [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: -4420849921117327522 time 0.070656 [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: -3946921629105938337 time 0.084992 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.059392 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.04608 [05/23/2020-11:20:16] [V] [TRT] Tactic: 1 time 0.064512 [05/23/2020-11:20:16] [V] [TRT] Tactic: 2 time 0.088064 [05/23/2020-11:20:16] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:20:16] [V] [TRT] Tactic: 5 time 0.172032 [05/23/2020-11:20:16] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.04608 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:20:16] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:20:16] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:20:16] [V] [TRT] [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.009216 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.009216 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:20:16] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:20:16] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:20:16] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:20:16] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:20:16] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:20:16] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: 1825138533642645384 time 0.265216 [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: 3915320020053085238 time 0.262144 [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: 6808617066150061604 time 0.162816 [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: -8060443123034038864 time 0.172032 [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: -4420849921117327522 time 0.191488 [05/23/2020-11:20:16] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:20:16] [V] [TRT] Tactic: -3946921629105938337 time 0.221184 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 6808617066150061604 Time: 0.162816 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.110592 [05/23/2020-11:20:16] [V] [TRT] Tactic: 1 time 0.166912 [05/23/2020-11:20:16] [V] [TRT] Tactic: 2 time 0.145408 [05/23/2020-11:20:16] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:20:16] [V] [TRT] Tactic: 5 time 0.36864 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.110592 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:20:16] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:20:16] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:20:16] [V] [TRT] [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.008224 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.008224 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.008224 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.008224 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:20:16] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:20:16] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:20:16] [V] [TRT] Tactic: 1 time 0.006176 [05/23/2020-11:20:16] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 1 Time: 0.006176 [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:16] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:20:16] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:20:16] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:16] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:17] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:17] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:17] [V] [TRT] Formats and tactics selection completed in 1.08135 seconds. [05/23/2020-11:20:17] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:20:17] [V] [TRT] Block size 1073741824 [05/23/2020-11:20:17] [V] [TRT] Block size 153600 [05/23/2020-11:20:17] [V] [TRT] Block size 153600 [05/23/2020-11:20:17] [V] [TRT] Block size 2048 [05/23/2020-11:20:17] [V] [TRT] Block size 2048 [05/23/2020-11:20:17] [V] [TRT] Block size 2048 [05/23/2020-11:20:17] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:20:17] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:20:17] [V] [TRT] Engine generation completed in 40.1046 seconds. [05/23/2020-11:20:17] [V] [TRT] Engine Layer Information: [05/23/2020-11:20:17] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:20:17] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:20:17] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:20:17] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:20:17] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:20:17] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:20:17] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:20:17] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:20:17] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:20:17] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:20:17] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:20:17] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:20:17] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:20:17] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:20:17] [V] [TRT] Original: 48 layers [05/23/2020-11:20:17] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:20:17] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:20:17] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:20:17] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:20:17] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:20:17] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:20:17] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:20:17] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:20:17] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:20:17] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:20:17] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:20:17] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:20:17] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:20:17] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:20:17] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:20:17] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:20:17] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:20:17] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:20:17] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:20:17] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:20:17] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:20:17] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:20:17] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:20:17] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:20:17] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:20:17] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:20:17] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:20:17] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:20:17] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:20:17] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:20:17] [V] [TRT] Graph construction and optimization completed in 0.0196495 seconds. [05/23/2020-11:20:17] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:20:17] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:17] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:20:17] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:17] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:20:17] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:17] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:17] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:17] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:17] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:17] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:20:17] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:17] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:20:17] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:20:17] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:20:17] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:20:17] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:17] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:17] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:20:17] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:20:17] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:20:17] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:20:17] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:20:18] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:20:18] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:20:18] [V] [TRT] Tactic: 2842488832350522458 time 0.016384 [05/23/2020-11:20:18] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:20:18] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:20:18] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:20:18] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:20:18] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:20:18] [V] [TRT] Tactic: 6808617066150061604 time 0.01536 [05/23/2020-11:20:18] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:20:18] [V] [TRT] Tactic: -8060443123034038864 time 0.016384 [05/23/2020-11:20:18] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:20:18] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:20:18] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:20:18] [V] [TRT] Tactic: -3946921629105938337 time 0.01536 [05/23/2020-11:20:18] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:20:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:20:18] [V] [TRT] Tactic: 0 time 0.01024 [05/23/2020-11:20:18] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:20:18] [V] [TRT] Tactic: 2 time 0.014336 [05/23/2020-11:20:18] [V] [TRT] Tactic: 4 time 1.59024 [05/23/2020-11:20:18] [V] [TRT] Tactic: 5 time 0.034816 [05/23/2020-11:20:18] [V] [TRT] Fastest Tactic: 0 Time: 0.01024 [05/23/2020-11:20:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:20:18] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:20:18] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:20:18] [V] [TRT] [05/23/2020-11:20:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:18] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:20:18] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:20:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:18] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:18] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:18] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:18] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:20:18] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:20:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:20:18] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:20:18] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:20:18] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:20:18] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:20:18] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:20:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:20:19] [V] [TRT] Tactic: 1 time 0.006112 [05/23/2020-11:20:19] [V] [TRT] Tactic: 2 time 0.00608 [05/23/2020-11:20:19] [V] [TRT] Fastest Tactic: 2 Time: 0.00608 [05/23/2020-11:20:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:20:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:19] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:20:19] [V] [TRT] Tactic: 0 time 0.006048 [05/23/2020-11:20:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006048 [05/23/2020-11:20:19] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:20:19] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:19] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:19] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:20:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:20:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:19] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:20:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:20:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:19] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:20:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:19] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:20:19] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:19] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:19] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:19] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:20:19] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:20:19] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:20:19] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:20:19] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:20:19] [V] [TRT] Tactic: -32 time 0.00912 [05/23/2020-11:20:19] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:20:19] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:20:19] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:20:19] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:20:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:20:20] [V] [TRT] Tactic: 1 time 0.008096 [05/23/2020-11:20:20] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:20:20] [V] [TRT] Tactic: 3 time 0.009216 [05/23/2020-11:20:20] [V] [TRT] Tactic: 6 time 0.050176 [05/23/2020-11:20:20] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:20:20] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:20:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:20:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:20:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:20:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:20] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:20:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:20:20] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:20] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:20:20] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:20:20] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:20:20] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:20:20] [V] [TRT] Tactic: 512 time 0.006048 [05/23/2020-11:20:20] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:20:20] [V] [TRT] Tactic: -64 time 0.007168 [05/23/2020-11:20:20] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:20:20] [V] [TRT] Fastest Tactic: 512 Time: 0.006048 [05/23/2020-11:20:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:20:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:20:20] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:20:20] [V] [TRT] Tactic: 3 time 0.009216 [05/23/2020-11:20:20] [V] [TRT] Tactic: 6 time 0.103424 [05/23/2020-11:20:20] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:20:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:20:20] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:20:20] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:20:20] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:20:20] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:20:20] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:20:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:20:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:21] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:20:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:21] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:20:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:20:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:21] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:20:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:20:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:21] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:20:21] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:21] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:20:21] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:20:21] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:20:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:20:21] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:20:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:21] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:21] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:20:21] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:20:21] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:20:21] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:20:21] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:20:21] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:20:21] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:20:21] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:20:22] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:22] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:22] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:22] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:20:22] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:20:22] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:20:22] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:20:22] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:20:22] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:20:22] [V] [TRT] Formats and tactics selection completed in 4.82763 seconds. [05/23/2020-11:20:22] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:20:22] [V] [TRT] Block size 1073741824 [05/23/2020-11:20:22] [V] [TRT] Block size 38400 [05/23/2020-11:20:22] [V] [TRT] Block size 38400 [05/23/2020-11:20:22] [V] [TRT] Block size 4608 [05/23/2020-11:20:22] [V] [TRT] Block size 2560 [05/23/2020-11:20:22] [V] [TRT] Block size 1024 [05/23/2020-11:20:22] [V] [TRT] Block size 1024 [05/23/2020-11:20:22] [V] [TRT] Block size 0 [05/23/2020-11:20:22] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:20:22] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:20:22] [V] [TRT] Engine generation completed in 5.1602 seconds. [05/23/2020-11:20:22] [V] [TRT] Engine Layer Information: [05/23/2020-11:20:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:20:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:20:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:20:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:20:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:20:22] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:20:22] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:20:22] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:20:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:20:22] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:20:22] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:20:22] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:20:22] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:20:22] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:20:22] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:20:22] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:20:22] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:20:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:20:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:20:22] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:20:22] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:20:22] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:20:22] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 512, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:20:22] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:20:22] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:20:22] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:20:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:20:22] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:20:22] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:20:22] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:20:22] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:20:22] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:20:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:20:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:20:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:20:22] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:20:22] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:20:22] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:20:22] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:20:22] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:20:22] [V] [TRT] Original: 12 layers [05/23/2020-11:20:22] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:20:22] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:20:22] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:20:22] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:20:22] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:20:22] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:20:22] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:20:22] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:20:22] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:20:22] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:20:22] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:20:22] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:20:22] [V] [TRT] Graph construction and optimization completed in 0.0407755 seconds. [05/23/2020-11:20:22] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:20:22] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:20:22] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:20:22] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:20:22] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:22] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:22] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:23] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:20:23] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:23] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:23] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:23] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:20:23] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:20:23] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:20:23] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:20:23] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:20:23] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:20:23] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:20:23] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:23] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:23] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:23] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:20:23] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:20:23] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:20:23] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:20:23] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:20:23] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:20:23] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:20:23] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:20:23] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:20:23] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:20:23] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:20:23] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:20:23] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:20:23] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:20:23] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:20:23] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:20:23] [V] [TRT] Formats and tactics selection completed in 0.93615 seconds. [05/23/2020-11:20:23] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:20:23] [V] [TRT] Block size 1073741824 [05/23/2020-11:20:23] [V] [TRT] Block size 512 [05/23/2020-11:20:23] [V] [TRT] Block size 512 [05/23/2020-11:20:23] [V] [TRT] Block size 512 [05/23/2020-11:20:23] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:20:23] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:20:24] [V] [TRT] Engine generation completed in 1.19109 seconds. [05/23/2020-11:20:24] [V] [TRT] Engine Layer Information: [05/23/2020-11:20:24] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:20:24] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:20:24] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:20:24] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:20:24] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:20:24] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:20:24] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:20:24] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:20:24] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:20:24] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:20:24] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:24] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:25] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:26] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:20:27] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread1 load float count:3834 thread2 load float count:3834 thread3 load float count:3834 thread4 load float count:3834 thread5 load float count:3834 thread6 load float count:3834 thread7 load float count:3834 thread8 load float count:3834 thread17 load float count:3834 thread12 load float count:3834 thread13 load float count:3834 thread11 load float count:3834 thread10 load float count:3834 thread9 load float count:3834 thread16 load float count:3834 thread15 load float count:3834 thread14 load float count:3834 thread18 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:21:06] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:21:06] [V] [TRT] Original: 18 layers [05/23/2020-11:21:06] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:21:06] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:21:06] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:21:06] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:21:06] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:21:06] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:21:06] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:21:06] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:21:06] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:21:06] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:21:06] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:21:06] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:21:06] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:21:06] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:21:06] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:21:06] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:21:06] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:21:06] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:21:06] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:21:06] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:21:06] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:21:06] [V] [TRT] Graph construction and optimization completed in 0.0201141 seconds. [05/23/2020-11:21:38] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:21:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:21:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:38] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:21:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:21:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:38] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:38] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:38] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:38] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:21:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:21:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:38] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:21:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:21:38] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:38] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:38] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:21:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:21:38] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:21:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:21:38] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:21:38] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:21:38] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:21:38] [V] [TRT] Tactic: 1825138533642645384 time 0.082944 [05/23/2020-11:21:38] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:21:39] [V] [TRT] Tactic: 3915320020053085238 time 0.08192 [05/23/2020-11:21:39] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:21:39] [V] [TRT] Tactic: 6808617066150061604 time 0.051264 [05/23/2020-11:21:39] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:21:39] [V] [TRT] Tactic: -8060443123034038864 time 0.054272 [05/23/2020-11:21:39] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:21:39] [V] [TRT] Tactic: -4420849921117327522 time 0.049152 [05/23/2020-11:21:39] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:21:39] [V] [TRT] Tactic: -3946921629105938337 time 0.06144 [05/23/2020-11:21:39] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.049152 [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:21:39] [V] [TRT] Tactic: 0 time 0.036864 [05/23/2020-11:21:39] [V] [TRT] Tactic: 1 time 0.060416 [05/23/2020-11:21:39] [V] [TRT] Tactic: 2 time 0.384096 [05/23/2020-11:21:39] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:21:39] [V] [TRT] Tactic: 5 time 0.546816 [05/23/2020-11:21:39] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:21:39] [V] [TRT] Fastest Tactic: 0 Time: 0.036864 [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:21:39] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:21:39] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:21:39] [V] [TRT] [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:39] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:21:39] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:39] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:21:39] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:39] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:21:39] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:39] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:39] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:39] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:39] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:39] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:21:39] [V] [TRT] Tactic: 1 time 0.00624 [05/23/2020-11:21:39] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:21:39] [V] [TRT] Fastest Tactic: 1 Time: 0.00624 [05/23/2020-11:21:39] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:21:39] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:21:39] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:21:39] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:39] [V] [TRT] Tactic: 0 time 0.007104 [05/23/2020-11:21:39] [V] [TRT] Fastest Tactic: 0 Time: 0.007104 [05/23/2020-11:21:39] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:21:39] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:21:39] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:21:39] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:21:39] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:21:39] [V] [TRT] Tactic: 1825138533642645384 time 0.263168 [05/23/2020-11:21:39] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:21:39] [V] [TRT] Tactic: 3915320020053085238 time 0.26112 [05/23/2020-11:21:39] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:21:39] [V] [TRT] Tactic: 6808617066150061604 time 0.1536 [05/23/2020-11:21:39] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:21:39] [V] [TRT] Tactic: -8060443123034038864 time 0.16384 [05/23/2020-11:21:39] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:21:39] [V] [TRT] Tactic: -4420849921117327522 time 0.146432 [05/23/2020-11:21:39] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:21:40] [V] [TRT] Tactic: -3946921629105938337 time 0.18432 [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.146432 [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:21:40] [V] [TRT] Tactic: 0 time 0.098304 [05/23/2020-11:21:40] [V] [TRT] Tactic: 1 time 0.160768 [05/23/2020-11:21:40] [V] [TRT] Tactic: 2 time 0.111616 [05/23/2020-11:21:40] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:21:40] [V] [TRT] Tactic: 5 time 0.4096 [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 0 Time: 0.098304 [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:21:40] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:21:40] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:21:40] [V] [TRT] [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:40] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:40] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:40] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:40] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:40] [V] [TRT] Tactic: 0 time 0.005216 [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 0 Time: 0.005216 [05/23/2020-11:21:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:21:40] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:21:40] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:21:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:21:40] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:21:40] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:40] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:21:40] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:21:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:40] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:21:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:40] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:21:40] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:21:40] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:40] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:40] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:40] [V] [TRT] Formats and tactics selection completed in 2.43828 seconds. [05/23/2020-11:21:40] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:21:40] [V] [TRT] Block size 1073741824 [05/23/2020-11:21:40] [V] [TRT] Block size 153600 [05/23/2020-11:21:40] [V] [TRT] Block size 153600 [05/23/2020-11:21:40] [V] [TRT] Block size 2048 [05/23/2020-11:21:40] [V] [TRT] Block size 2048 [05/23/2020-11:21:40] [V] [TRT] Block size 2048 [05/23/2020-11:21:40] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:21:40] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:21:41] [V] [TRT] Engine generation completed in 34.4349 seconds. [05/23/2020-11:21:41] [V] [TRT] Engine Layer Information: [05/23/2020-11:21:41] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:21:41] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:21:41] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:21:41] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:21:41] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:21:41] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:21:41] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:21:41] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:21:41] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:21:41] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:21:41] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:21:41] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:21:41] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:21:41] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:21:41] [V] [TRT] Original: 48 layers [05/23/2020-11:21:41] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:21:41] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:21:41] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:21:41] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:21:41] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:21:41] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:21:41] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:21:41] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:21:41] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:21:41] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:21:41] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:21:41] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:21:41] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:21:41] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:21:41] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:21:41] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:21:41] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:21:41] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:21:41] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:21:41] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:21:41] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:21:41] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:21:41] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:21:41] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:21:41] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:21:41] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:21:41] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:21:41] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:21:41] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:21:41] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:21:41] [V] [TRT] Graph construction and optimization completed in 0.314002 seconds. [05/23/2020-11:21:41] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:21:41] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:21:41] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:21:41] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:21:41] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:21:41] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:21:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:42] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:21:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:42] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:21:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:42] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:42] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:42] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:21:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:42] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:21:42] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:21:42] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:21:42] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:21:42] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:42] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:21:42] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:21:42] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:21:42] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:21:42] [V] [TRT] Tactic: 1825138533642645384 time 0.019456 [05/23/2020-11:21:42] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:21:42] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:21:42] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:21:42] [V] [TRT] Tactic: 3915320020053085238 time 0.017408 [05/23/2020-11:21:42] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:21:42] [V] [TRT] Tactic: 6448355332020552203 time 0.018432 [05/23/2020-11:21:42] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:21:42] [V] [TRT] Tactic: 6808617066150061604 time 0.01536 [05/23/2020-11:21:42] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:21:42] [V] [TRT] Tactic: -8060443123034038864 time 0.016384 [05/23/2020-11:21:42] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:21:42] [V] [TRT] Tactic: -4420849921117327522 time 0.013312 [05/23/2020-11:21:42] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:21:42] [V] [TRT] Tactic: -3946921629105938337 time 0.01536 [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.013312 [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:21:42] [V] [TRT] Tactic: 0 time 0.01024 [05/23/2020-11:21:42] [V] [TRT] Tactic: 1 time 0.017472 [05/23/2020-11:21:42] [V] [TRT] Tactic: 2 time 0.014368 [05/23/2020-11:21:42] [V] [TRT] Tactic: 4 time 1.59642 [05/23/2020-11:21:42] [V] [TRT] Tactic: 5 time 0.033792 [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: 0 Time: 0.01024 [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:21:42] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:21:42] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:21:42] [V] [TRT] [05/23/2020-11:21:42] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:42] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:42] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 time 0.00624 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0.00624 [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:21:43] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:21:43] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:21:43] [V] [TRT] Tactic: 2 time 0.00512 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:21:43] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:21:43] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:21:43] [V] [TRT] Tactic: 512 time 0.0072 [05/23/2020-11:21:43] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:21:43] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:21:43] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 256 Time: 0.007168 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:21:43] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:21:43] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:21:43] [V] [TRT] Tactic: 6 time 0.051264 [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:21:43] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:43] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:43] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:21:43] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:21:43] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:21:43] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:21:44] [V] [TRT] Tactic: -64 time 0.0072 [05/23/2020-11:21:44] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:21:44] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:21:44] [V] [TRT] Tactic: 3 time 0.009216 [05/23/2020-11:21:44] [V] [TRT] Tactic: 6 time 0.1024 [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:21:44] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:21:44] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:21:44] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:44] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:21:44] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:21:44] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:21:44] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:21:44] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:21:44] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:44] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:21:44] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:21:44] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:44] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:44] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:21:44] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:21:44] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:21:44] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:21:44] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:21:44] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:21:45] [V] [TRT] Formats and tactics selection completed in 3.23501 seconds. [05/23/2020-11:21:45] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:21:45] [V] [TRT] Block size 1073741824 [05/23/2020-11:21:45] [V] [TRT] Block size 38400 [05/23/2020-11:21:45] [V] [TRT] Block size 38400 [05/23/2020-11:21:45] [V] [TRT] Block size 4608 [05/23/2020-11:21:45] [V] [TRT] Block size 2560 [05/23/2020-11:21:45] [V] [TRT] Block size 1024 [05/23/2020-11:21:45] [V] [TRT] Block size 1024 [05/23/2020-11:21:45] [V] [TRT] Block size 0 [05/23/2020-11:21:45] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:21:45] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:21:45] [V] [TRT] Engine generation completed in 3.56868 seconds. [05/23/2020-11:21:45] [V] [TRT] Engine Layer Information: [05/23/2020-11:21:45] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:21:45] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:21:45] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:21:45] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:21:45] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:21:45] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:21:45] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:21:45] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:21:45] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:21:45] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:21:45] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:21:45] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:21:45] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:21:45] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:21:45] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:21:45] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:21:45] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:21:45] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:21:45] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:21:45] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:21:45] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:21:45] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:21:45] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:21:45] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:21:45] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:21:45] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:21:45] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:21:45] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:21:45] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:21:45] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:21:45] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:21:45] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:21:45] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:21:45] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:21:45] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:21:45] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:21:45] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:21:45] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:21:45] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:21:45] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:21:45] [V] [TRT] Original: 12 layers [05/23/2020-11:21:45] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:21:45] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:21:45] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:21:45] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:21:45] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:21:45] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:21:45] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:21:45] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:21:45] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:21:45] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:21:45] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:21:45] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:21:45] [V] [TRT] Graph construction and optimization completed in 0.173247 seconds. [05/23/2020-11:21:45] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:21:45] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:21:45] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:21:45] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:21:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:45] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:21:45] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:45] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:45] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:45] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:21:45] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:21:45] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:21:45] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:21:45] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:21:45] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:21:45] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:21:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:46] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:21:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:21:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:21:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:21:46] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:21:46] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:21:46] [V] [TRT] Tactic: 128 time 0.007168 [05/23/2020-11:21:46] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:21:46] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:21:46] [V] [TRT] Fastest Tactic: 256 Time: 0.006144 [05/23/2020-11:21:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:21:46] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:21:46] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:21:46] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:21:46] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:21:46] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:21:46] [V] [TRT] Formats and tactics selection completed in 0.91854 seconds. [05/23/2020-11:21:46] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:21:46] [V] [TRT] Block size 1073741824 [05/23/2020-11:21:46] [V] [TRT] Block size 512 [05/23/2020-11:21:46] [V] [TRT] Block size 512 [05/23/2020-11:21:46] [V] [TRT] Block size 512 [05/23/2020-11:21:46] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:21:46] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:21:46] [V] [TRT] Engine generation completed in 1.30814 seconds. [05/23/2020-11:21:46] [V] [TRT] Engine Layer Information: [05/23/2020-11:21:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:21:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:21:46] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:21:46] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:21:46] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:21:46] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:21:46] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:21:46] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:21:46] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 256, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:21:46] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:21:46] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:47] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:48] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:21:49] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread3 load float count:3834 thread5 load float count:3834 thread8 load float count:3834 thread12 load float count:3834 thread1 load float count:3834 thread17 load float count:3834 thread11 load float count:3834 thread2 load float count:3834 thread6 load float count:3834 thread10 load float count:3834 thread0 load float count:3834 thread19 load float count:3834 thread4 load float count:3834 thread16 load float count:3834 thread9 load float count:3834 thread13 load float count:3834 thread18 load float count:3834 thread15 load float count:3834 thread14 load float count:3834 thread7 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:22:25] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:22:25] [V] [TRT] Original: 18 layers [05/23/2020-11:22:25] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:22:25] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:22:25] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:22:25] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:22:25] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:22:25] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:22:25] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:22:25] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:22:25] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:22:25] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:22:25] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:22:25] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:22:25] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:22:25] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:22:25] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:22:25] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:22:25] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:22:25] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:22:25] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:22:25] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:22:25] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:22:25] [V] [TRT] Graph construction and optimization completed in 0.0478631 seconds. [05/23/2020-11:22:46] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:22:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:22:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:46] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:22:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:22:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:46] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:46] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:46] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:46] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:22:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:22:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:46] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:22:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:22:46] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:46] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:46] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:22:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:22:46] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:22:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:22:46] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:22:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:22:46] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:22:46] [V] [TRT] Tactic: 1825138533642645384 time 0.083968 [05/23/2020-11:22:46] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:22:46] [V] [TRT] Tactic: 3915320020053085238 time 0.082944 [05/23/2020-11:22:46] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:22:46] [V] [TRT] Tactic: 6808617066150061604 time 0.052224 [05/23/2020-11:22:46] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:22:46] [V] [TRT] Tactic: -8060443123034038864 time 0.055296 [05/23/2020-11:22:46] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:22:46] [V] [TRT] Tactic: -4420849921117327522 time 0.050176 [05/23/2020-11:22:46] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:22:46] [V] [TRT] Tactic: -3946921629105938337 time 0.062464 [05/23/2020-11:22:46] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.050176 [05/23/2020-11:22:46] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:22:46] [V] [TRT] Tactic: 0 time 0.037888 [05/23/2020-11:22:47] [V] [TRT] Tactic: 1 time 0.060416 [05/23/2020-11:22:47] [V] [TRT] Tactic: 2 time 0.067584 [05/23/2020-11:22:47] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:22:47] [V] [TRT] Tactic: 5 time 0.144384 [05/23/2020-11:22:47] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 0 Time: 0.037888 [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:22:47] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:22:47] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:22:47] [V] [TRT] [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:47] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:47] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:47] [V] [TRT] Tactic: 0 time 0.006176 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 0 Time: 0.006176 [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:47] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:47] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:47] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:22:47] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:22:47] [V] [TRT] Tactic: 2 time 0.009216 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:22:47] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:22:47] [V] [TRT] Tactic: 1 time 0.007136 [05/23/2020-11:22:47] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 1 Time: 0.007136 [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:47] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:22:47] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:22:47] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:22:47] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:22:47] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:22:47] [V] [TRT] Tactic: 1825138533642645384 time 0.26112 [05/23/2020-11:22:47] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:22:47] [V] [TRT] Tactic: 3915320020053085238 time 0.259072 [05/23/2020-11:22:47] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:22:47] [V] [TRT] Tactic: 6808617066150061604 time 0.151552 [05/23/2020-11:22:47] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:22:47] [V] [TRT] Tactic: -8060443123034038864 time 0.161792 [05/23/2020-11:22:47] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:22:47] [V] [TRT] Tactic: -4420849921117327522 time 0.145408 [05/23/2020-11:22:47] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:22:47] [V] [TRT] Tactic: -3946921629105938337 time 0.183296 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.145408 [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:22:47] [V] [TRT] Tactic: 0 time 0.09728 [05/23/2020-11:22:47] [V] [TRT] Tactic: 1 time 0.15872 [05/23/2020-11:22:47] [V] [TRT] Tactic: 2 time 0.109568 [05/23/2020-11:22:47] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:22:47] [V] [TRT] Tactic: 5 time 0.64 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 0 Time: 0.09728 [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:22:47] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:22:47] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:22:47] [V] [TRT] [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:47] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:22:47] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:47] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:22:47] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:22:48] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:48] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:22:48] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:22:48] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:48] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:48] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:48] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:48] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:48] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:48] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:22:48] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:22:48] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:22:48] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:22:48] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:22:48] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:22:48] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:22:48] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:22:48] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:22:48] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:22:48] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:48] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:22:48] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:22:48] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:22:48] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:22:48] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:48] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:48] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:22:48] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:22:48] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:48] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:48] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:22:48] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:22:48] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:48] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:48] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:48] [V] [TRT] Formats and tactics selection completed in 2.19324 seconds. [05/23/2020-11:22:48] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:22:48] [V] [TRT] Block size 1073741824 [05/23/2020-11:22:48] [V] [TRT] Block size 153600 [05/23/2020-11:22:48] [V] [TRT] Block size 153600 [05/23/2020-11:22:48] [V] [TRT] Block size 2048 [05/23/2020-11:22:48] [V] [TRT] Block size 2048 [05/23/2020-11:22:48] [V] [TRT] Block size 2048 [05/23/2020-11:22:48] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:22:48] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:22:48] [V] [TRT] Engine generation completed in 23.027 seconds. [05/23/2020-11:22:48] [V] [TRT] Engine Layer Information: [05/23/2020-11:22:48] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:22:48] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:22:48] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:22:48] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:22:48] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:22:48] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:22:48] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:22:48] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:22:48] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:22:48] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:22:48] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:22:48] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:22:48] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:22:48] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:22:48] [V] [TRT] Original: 48 layers [05/23/2020-11:22:48] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:22:48] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:22:48] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:22:48] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:22:48] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:22:48] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:22:48] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:22:48] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:22:48] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:22:48] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:22:48] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:22:48] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:22:48] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:22:48] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:22:48] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:22:48] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:22:48] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:22:48] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:22:48] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:22:48] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:22:48] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:22:48] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:22:48] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:22:48] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:22:48] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:22:48] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:22:48] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:22:48] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:22:48] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:22:48] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:22:48] [V] [TRT] Graph construction and optimization completed in 0.134052 seconds. [05/23/2020-11:22:49] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:22:49] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:22:49] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:22:49] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:22:49] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:22:49] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:22:49] [V] [TRT] Tactic: 1825138533642645384 time 0.018432 [05/23/2020-11:22:49] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:22:49] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:22:49] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:22:49] [V] [TRT] Tactic: 3915320020053085238 time 0.018432 [05/23/2020-11:22:49] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:22:49] [V] [TRT] Tactic: 6448355332020552203 time 0.019456 [05/23/2020-11:22:49] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:22:49] [V] [TRT] Tactic: 6808617066150061604 time 0.01536 [05/23/2020-11:22:49] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:22:49] [V] [TRT] Tactic: -8060443123034038864 time 0.017408 [05/23/2020-11:22:49] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:22:49] [V] [TRT] Tactic: -4420849921117327522 time 0.014336 [05/23/2020-11:22:49] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:22:49] [V] [TRT] Tactic: -3946921629105938337 time 0.016384 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.014336 [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 time 0.011264 [05/23/2020-11:22:49] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:22:49] [V] [TRT] Tactic: 2 time 0.01632 [05/23/2020-11:22:49] [V] [TRT] Tactic: 4 time 1.61792 [05/23/2020-11:22:49] [V] [TRT] Tactic: 5 time 0.036896 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0.011264 [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:22:49] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:22:49] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:22:49] [V] [TRT] [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 time 0.006208 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0.006208 [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:49] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:22:49] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:22:49] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:22:49] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:22:49] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:22:49] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:22:49] [V] [TRT] Tactic: 2 time 0.00512 [05/23/2020-11:22:49] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:22:50] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:22:50] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:22:50] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:22:50] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:22:50] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:50] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:50] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:22:50] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:22:50] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:22:50] [V] [TRT] Tactic: 512 time 0.007168 [05/23/2020-11:22:50] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:22:50] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:22:50] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 256 Time: 0.007168 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:22:50] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:22:50] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:22:50] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:22:50] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 2 Time: 0.006144 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:22:50] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:22:50] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:22:50] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:22:50] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:22:50] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:22:50] [V] [TRT] Tactic: 512 time 0.005216 [05/23/2020-11:22:50] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:22:50] [V] [TRT] Tactic: -64 time 0.007232 [05/23/2020-11:22:50] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 512 Time: 0.005216 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:22:50] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:22:50] [V] [TRT] Tactic: 3 time 0.009216 [05/23/2020-11:22:50] [V] [TRT] Tactic: 6 time 0.103424 [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:22:50] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:22:50] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:22:50] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:22:50] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:22:50] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:22:51] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:22:51] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:51] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:51] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:22:51] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:51] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:51] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:51] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:22:51] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:22:51] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:22:51] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:22:51] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:22:51] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:22:51] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:22:51] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:22:51] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:51] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:51] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:22:51] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:22:51] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:22:51] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:22:51] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:22:51] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:22:51] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:22:51] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:22:51] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:51] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:51] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:51] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:22:51] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:22:51] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:22:51] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:22:51] [V] [TRT] Tactic: 2 time 0.00512 [05/23/2020-11:22:51] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:22:51] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:22:51] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:22:51] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:51] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:51] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:52] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:22:52] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:22:52] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:22:52] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:22:52] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:22:52] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:22:52] [V] [TRT] Formats and tactics selection completed in 3.2009 seconds. [05/23/2020-11:22:52] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:22:52] [V] [TRT] Block size 1073741824 [05/23/2020-11:22:52] [V] [TRT] Block size 38400 [05/23/2020-11:22:52] [V] [TRT] Block size 38400 [05/23/2020-11:22:52] [V] [TRT] Block size 4608 [05/23/2020-11:22:52] [V] [TRT] Block size 2560 [05/23/2020-11:22:52] [V] [TRT] Block size 1024 [05/23/2020-11:22:52] [V] [TRT] Block size 1024 [05/23/2020-11:22:52] [V] [TRT] Block size 0 [05/23/2020-11:22:52] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:22:52] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:22:52] [V] [TRT] Engine generation completed in 3.7609 seconds. [05/23/2020-11:22:52] [V] [TRT] Engine Layer Information: [05/23/2020-11:22:52] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:22:52] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:22:52] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:22:52] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:22:52] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:22:52] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:22:52] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:22:52] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:22:52] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:22:52] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:22:52] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:22:52] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:22:52] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:22:52] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:22:52] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:22:52] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:22:52] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:22:52] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:22:52] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:22:52] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:22:52] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:22:52] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:22:52] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 512, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:22:52] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:22:52] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:22:52] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:22:52] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:22:52] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:22:52] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:22:52] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:22:52] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:22:52] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:22:52] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:22:52] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:22:52] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:22:52] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:22:52] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:22:52] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:22:52] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:22:52] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:22:52] [V] [TRT] Original: 12 layers [05/23/2020-11:22:52] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:22:52] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:22:52] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:22:52] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:22:52] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:22:52] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:22:52] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:22:52] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:22:52] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:22:52] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:22:52] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:22:52] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:22:52] [V] [TRT] Graph construction and optimization completed in 0.0631387 seconds. [05/23/2020-11:22:52] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:22:52] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:22:52] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:22:52] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:22:52] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:52] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:52] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:52] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:22:52] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:52] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:52] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:52] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:22:52] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:22:52] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:22:52] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:22:52] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:22:52] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:22:53] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:22:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:53] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:53] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:53] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:22:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:22:53] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:22:53] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:22:53] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:22:53] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:22:53] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:22:53] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:22:53] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:22:53] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:22:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:22:53] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:22:53] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:22:53] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:22:53] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:22:53] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:22:53] [V] [TRT] Formats and tactics selection completed in 0.807001 seconds. [05/23/2020-11:22:53] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:22:53] [V] [TRT] Block size 1073741824 [05/23/2020-11:22:53] [V] [TRT] Block size 512 [05/23/2020-11:22:53] [V] [TRT] Block size 512 [05/23/2020-11:22:53] [V] [TRT] Block size 512 [05/23/2020-11:22:53] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:22:53] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:22:53] [V] [TRT] Engine generation completed in 0.859319 seconds. [05/23/2020-11:22:53] [V] [TRT] Engine Layer Information: [05/23/2020-11:22:53] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:22:53] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:22:53] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:22:53] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:22:53] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:22:53] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:22:53] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:22:53] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:22:53] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:22:53] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:22:53] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:53] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:54] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:22:55] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread2 load float count:3834 thread4 load float count:3834 thread6 load float count:3834 thread8 load float count:3834 thread10 load float count:3834 thread0 load float count:3834 thread1 load float count:3834 thread5 load float count:3834 thread12 load float count:3834 thread9 load float count:3834 thread3 load float count:3834 thread7 load float count:3834 thread11 load float count:3834 thread13 load float count:3834 thread17 load float count:3834 thread14 load float count:3834 thread15 load float count:3834 thread18 load float count:3834 thread16 load float count:3834 thread19 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:23:31] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:23:31] [V] [TRT] Original: 18 layers [05/23/2020-11:23:31] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:23:31] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:23:31] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:23:31] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:23:31] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:23:31] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:23:31] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:23:31] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:23:31] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:23:31] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:23:31] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:23:31] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:23:31] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:23:31] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:23:31] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:23:31] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:23:31] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:23:31] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:23:31] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:23:31] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:23:31] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:23:31] [V] [TRT] Graph construction and optimization completed in 0.053078 seconds. [05/23/2020-11:23:51] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: Float(1,71,10650) -> Float(1,150,150,10650) *************** [05/23/2020-11:23:51] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Shuffle] (Shuffle) [05/23/2020-11:23:51] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:51] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:51] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:23:52] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Shuffle] (Shuffle) [05/23/2020-11:23:52] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:52] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:52] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:52] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:52] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:52] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:23:52] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Shuffle] (Shuffle) [05/23/2020-11:23:52] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:52] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:52] [V] [TRT] *************** Autotuning format combination: Float(1,512,512) -> Float(1,256,512) *************** [05/23/2020-11:23:52] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 15) [Shuffle] (Shuffle) [05/23/2020-11:23:52] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:52] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:52] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,10650) -> Float(1,150,150,38400) *************** [05/23/2020-11:23:52] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (LegacySASSConvolution) [05/23/2020-11:23:52] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:23:52] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (FusedConvActConvolution) [05/23/2020-11:23:52] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:23:52] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CaskConvolution) [05/23/2020-11:23:52] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:23:52] [V] [TRT] Tactic: 1825138533642645384 time 0.084992 [05/23/2020-11:23:52] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:23:52] [V] [TRT] Tactic: 3915320020053085238 time 0.083968 [05/23/2020-11:23:52] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:23:52] [V] [TRT] Tactic: 6808617066150061604 time 0.053248 [05/23/2020-11:23:52] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:23:52] [V] [TRT] Tactic: -8060443123034038864 time 0.05632 [05/23/2020-11:23:52] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:23:52] [V] [TRT] Tactic: -4420849921117327522 time 0.050176 [05/23/2020-11:23:52] [V] [TRT] (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:23:52] [V] [TRT] Tactic: -3946921629105938337 time 0.062496 [05/23/2020-11:23:52] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.050176 [05/23/2020-11:23:52] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaConvolution) [05/23/2020-11:23:52] [V] [TRT] Tactic: 0 time 0.037888 [05/23/2020-11:23:52] [V] [TRT] Tactic: 1 time 0.06144 [05/23/2020-11:23:52] [V] [TRT] Tactic: 2 time 0.068608 [05/23/2020-11:23:52] [V] [TRT] Tactic: 4 skipped. Scratch requested: 9642995712, available: 1073741824 [05/23/2020-11:23:52] [V] [TRT] Tactic: 5 time 0.151552 [05/23/2020-11:23:52] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [05/23/2020-11:23:52] [V] [TRT] Fastest Tactic: 0 Time: 0.037888 [05/23/2020-11:23:52] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:23:52] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:23:52] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:23:52] [V] [TRT] [05/23/2020-11:23:52] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:52] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:23:52] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:23:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:53] [V] [TRT] Tactic: 0 time 0.0072 [05/23/2020-11:23:53] [V] [TRT] Fastest Tactic: 0 Time: 0.0072 [05/23/2020-11:23:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:53] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:23:53] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:23:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:53] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:53] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:53] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:53] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:53] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:23:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:23:53] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:23:53] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:23:53] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:23:53] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:23:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise] (ElementWise) [05/23/2020-11:23:53] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:23:53] [V] [TRT] Tactic: 2 time 0.008192 [05/23/2020-11:23:53] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:23:53] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:53] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:23:53] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:23:53] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,150,150,38400) *************** [05/23/2020-11:23:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (LegacySASSConvolution) [05/23/2020-11:23:53] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:23:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (FusedConvActConvolution) [05/23/2020-11:23:53] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:23:53] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CaskConvolution) [05/23/2020-11:23:53] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:23:54] [V] [TRT] Tactic: 1825138533642645384 time 0.270336 [05/23/2020-11:23:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:23:54] [V] [TRT] Tactic: 3915320020053085238 time 0.268288 [05/23/2020-11:23:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:23:54] [V] [TRT] Tactic: 6808617066150061604 time 0.157696 [05/23/2020-11:23:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:23:54] [V] [TRT] Tactic: -8060443123034038864 time 0.167936 [05/23/2020-11:23:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:23:54] [V] [TRT] Tactic: -4420849921117327522 time 0.150528 [05/23/2020-11:23:54] [V] [TRT] (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:23:54] [V] [TRT] Tactic: -3946921629105938337 time 0.18944 [05/23/2020-11:23:54] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.150528 [05/23/2020-11:23:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaConvolution) [05/23/2020-11:23:54] [V] [TRT] Tactic: 0 time 0.101376 [05/23/2020-11:23:54] [V] [TRT] Tactic: 1 time 0.164864 [05/23/2020-11:23:54] [V] [TRT] Tactic: 2 time 0.113664 [05/23/2020-11:23:54] [V] [TRT] Tactic: 4 skipped. Scratch requested: 34765012992, available: 1073741824 [05/23/2020-11:23:54] [V] [TRT] Tactic: 5 time 0.36352 [05/23/2020-11:23:54] [V] [TRT] Fastest Tactic: 0 Time: 0.101376 [05/23/2020-11:23:54] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation] (CudaDepthwiseConvolution) [05/23/2020-11:23:54] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:23:54] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:23:54] [V] [TRT] [05/23/2020-11:23:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:54] [V] [TRT] Tactic: 0 time 0.008192 [05/23/2020-11:23:54] [V] [TRT] Fastest Tactic: 0 Time: 0.008192 [05/23/2020-11:23:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:23:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:23:54] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:54] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:23:54] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:23:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:55] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:55] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400), Float(1,150,150,150) -> Float(1,150,150,38400) *************** [05/23/2020-11:23:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:23:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:23:55] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:23:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:23:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,1200), Float(1,150,150:32,150) -> Float(1,150,150:32,1200) *************** [05/23/2020-11:23:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:23:55] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:23:55] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:23:55] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:23:55] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:55] [V] [TRT] Tactic: 0 time 0.007168 [05/23/2020-11:23:55] [V] [TRT] Fastest Tactic: 0 Time: 0.007168 [05/23/2020-11:23:55] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,38400) -> Float(1,256,38400) *************** [05/23/2020-11:23:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:23:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:55] [V] [TRT] *************** Autotuning format combination: Float(1,256,38400), Float(1,256,512), Float(1,256,512), Int32(1) -> Float(1,512,76800), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:23:55] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 13) [RNN] (RNNv2) [05/23/2020-11:23:55] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:55] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,32768) *************** [05/23/2020-11:23:56] [V] [TRT] *************** Autotuning format combination: Float(1,512,76800), Float(1,64,32768) -> Float(1,64,9600) *************** [05/23/2020-11:23:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 17) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:23:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:56] [V] [TRT] Formats and tactics selection completed in 4.14972 seconds. [05/23/2020-11:23:56] [V] [TRT] After reformat layers: 12 layers [05/23/2020-11:23:56] [V] [TRT] Block size 1073741824 [05/23/2020-11:23:56] [V] [TRT] Block size 153600 [05/23/2020-11:23:56] [V] [TRT] Block size 153600 [05/23/2020-11:23:56] [V] [TRT] Block size 2048 [05/23/2020-11:23:56] [V] [TRT] Block size 2048 [05/23/2020-11:23:56] [V] [TRT] Block size 2048 [05/23/2020-11:23:56] [V] [TRT] Total Activation Memory: 1074055168 [05/23/2020-11:23:56] [I] [TRT] Detected 5 inputs and 2 output network tensors. [05/23/2020-11:23:56] [V] [TRT] Engine generation completed in 24.8258 seconds. [05/23/2020-11:23:56] [V] [TRT] Engine Layer Information: [05/23/2020-11:23:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 0) [Shuffle], Tactic: 0, encoder-input-data[Float(150,71)] -> (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] [05/23/2020-11:23:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 1) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:23:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 14) [Shuffle], Tactic: 0, encoder-input-lstm-hidden[Float(1,512)] -> (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)] [05/23/2020-11:23:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 15) [Shuffle], Tactic: 0, encoder-input-lstm-cell[Float(1,512)] -> (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)] [05/23/2020-11:23:56] [V] [TRT] Layer(Convolution): (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] + (Unnamed Layer* 5) [Activation], Tactic: 0, (Unnamed Layer* 0) [Shuffle]_output[Float(71,1,150)] -> (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)] [05/23/2020-11:23:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 6) [ElementWise], Tactic: 1, (Unnamed Layer* 5) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:23:56] [V] [TRT] Layer(Convolution): (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] + (Unnamed Layer* 10) [Activation], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)] [05/23/2020-11:23:56] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Activation]_output[Float(256,1,150)], (Unnamed Layer* 1) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] [05/23/2020-11:23:56] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(256,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)] [05/23/2020-11:23:56] [V] [TRT] Layer(RNN): (Unnamed Layer* 13) [RNN], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,256)], (Unnamed Layer* 14) [Shuffle]_output[Float(2,256)], (Unnamed Layer* 15) [Shuffle]_output[Float(2,256)], actual-encoder-input-sequence-length[Int32()] -> encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 13) [RNN]_output_2[Float(2,256)], (Unnamed Layer* 13) [RNN]_output_3[Float(2,256)] [05/23/2020-11:23:56] [V] [TRT] Layer(Constant): (Unnamed Layer* 16) [Constant], Tactic: 0, -> (Unnamed Layer* 16) [Constant]_output[Float(512,64)] [05/23/2020-11:23:56] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 17) [Matrix Multiply], Tactic: 0, encoder-output-cat-embedding-data[Float(150,512)], (Unnamed Layer* 16) [Constant]_output[Float(512,64)] -> attention-keys[Float(150,64)] [05/23/2020-11:23:56] [V] [TRT] Bias weights are not set yet. Bias weights can be set using setInput(2, bias_tensor) API call. [05/23/2020-11:23:56] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:23:56] [V] [TRT] Original: 48 layers [05/23/2020-11:23:56] [V] [TRT] After dead-layer removal: 48 layers [05/23/2020-11:23:56] [V] [TRT] After Myelin optimization: 48 layers [05/23/2020-11:23:56] [V] [TRT] After scale fusion: 48 layers [05/23/2020-11:23:56] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:23:56] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:23:56] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:23:56] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:23:56] [V] [TRT] Fusing (Unnamed Layer* 9) [Padding] with (Unnamed Layer* 10) [Convolution] [05/23/2020-11:23:56] [V] [TRT] Fusing (Unnamed Layer* 3) [ElementWise] with (Unnamed Layer* 4) [Activation] [05/23/2020-11:23:56] [V] [TRT] Modifying configuration of (Unnamed Layer* 31) [Reduce] [05/23/2020-11:23:56] [V] [TRT] Fusing (Unnamed Layer* 41) [ElementWise] with (Unnamed Layer* 42) [Activation] [05/23/2020-11:23:56] [V] [TRT] Fusing (Unnamed Layer* 28) [ElementWise] with (Unnamed Layer* 30) [ElementWise] [05/23/2020-11:23:56] [V] [TRT] Fusing (Unnamed Layer* 17) [ElementWise] with (Unnamed Layer* 18) [ElementWise] [05/23/2020-11:23:56] [V] [TRT] Fusing PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]) with (Unnamed Layer* 19) [Activation] [05/23/2020-11:23:56] [V] [TRT] Fusing PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]) with (Unnamed Layer* 21) [ElementWise] [05/23/2020-11:23:56] [V] [TRT] Fusing (Unnamed Layer* 45) [Constant] with (Unnamed Layer* 46) [ElementWise] [05/23/2020-11:23:56] [V] [TRT] Fusing PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]) with (Unnamed Layer* 47) [Activation] [05/23/2020-11:23:56] [V] [TRT] After vertical fusions: 39 layers [05/23/2020-11:23:56] [V] [TRT] After final dead-layer removal: 39 layers [05/23/2020-11:23:56] [V] [TRT] After tensor merging: 39 layers [05/23/2020-11:23:56] [V] [TRT] Eliminating concatenation (Unnamed Layer* 5) [Concatenation] [05/23/2020-11:23:56] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:23:56] [V] [TRT] Generating copy for decoder-input-of-previous-attention-output to (Unnamed Layer* 5) [Concatenation]_output [05/23/2020-11:23:56] [V] [TRT] Eliminating concatenation (Unnamed Layer* 35) [Concatenation] [05/23/2020-11:23:56] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:23:56] [V] [TRT] Generating copy for (Unnamed Layer* 4) [Activation]_output to (Unnamed Layer* 35) [Concatenation]_output [05/23/2020-11:23:56] [V] [TRT] Eliminating concatenation (Unnamed Layer* 37) [Concatenation] [05/23/2020-11:23:56] [V] [TRT] Generating copy for decoder-output-attention to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:23:56] [V] [TRT] Generating copy for (Unnamed Layer* 36) [RNN]_output_1 to (Unnamed Layer* 37) [Concatenation]_output [05/23/2020-11:23:56] [V] [TRT] After concat removal: 42 layers [05/23/2020-11:23:56] [V] [TRT] Graph construction and optimization completed in 0.110713 seconds. [05/23/2020-11:23:56] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,640,25600) *************** [05/23/2020-11:23:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,640,640) *************** [05/23/2020-11:23:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,2048) *************** [05/23/2020-11:23:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,8192) *************** [05/23/2020-11:23:56] [V] [TRT] *************** Autotuning format combination: -> Float(1,64,64) *************** [05/23/2020-11:23:56] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:23:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 25) [Shuffle] (Shuffle) [05/23/2020-11:23:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:56] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:23:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Shuffle] (Shuffle) [05/23/2020-11:23:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:56] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150,150) *************** [05/23/2020-11:23:56] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Shuffle] (Shuffle) [05/23/2020-11:23:56] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:56] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:56] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:56] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:23:56] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:23:56] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,640,25600) -> Float(1,640,640) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 1) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,150) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 26) [Padding] (Padding) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,640,640), Float(1,640,640) -> Float(1,640,640) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation] (ElementWise) [05/23/2020-11:23:57] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:23:57] [V] [TRT] Tactic: 2 time 0.013312 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,1,150) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 27) [Shuffle] (Shuffle) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (LegacySASSConvolution) [05/23/2020-11:23:57] [V] [TRT] LegacySASSConvolution has no valid tactics for this config, skipping [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (FusedConvActConvolution) [05/23/2020-11:23:57] [V] [TRT] FusedConvActConvolution has no valid tactics for this config, skipping [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CaskConvolution) [05/23/2020-11:23:57] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_medium_nn_v1 [05/23/2020-11:23:57] [V] [TRT] Tactic: 1825138533642645384 time 0.018432 [05/23/2020-11:23:57] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_xregs_large_nn_v1 [05/23/2020-11:23:57] [V] [TRT] Tactic: 2842488832350522458 time 0.017408 [05/23/2020-11:23:57] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_small_nn_v1 [05/23/2020-11:23:57] [V] [TRT] Tactic: 3915320020053085238 time 0.017408 [05/23/2020-11:23:57] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x128_relu_xregs_large_nn_v1 [05/23/2020-11:23:57] [V] [TRT] Tactic: 6448355332020552203 time 0.018432 [05/23/2020-11:23:57] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_small_nn_v1 [05/23/2020-11:23:57] [V] [TRT] Tactic: 6808617066150061604 time 0.01536 [05/23/2020-11:23:57] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x64_relu_medium_nn_v1 [05/23/2020-11:23:57] [V] [TRT] Tactic: -8060443123034038864 time 0.016384 [05/23/2020-11:23:57] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_medium_nn_v1 [05/23/2020-11:23:57] [V] [TRT] Tactic: -4420849921117327522 time 0.013312 [05/23/2020-11:23:57] [V] [TRT] (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (scudnn) Set Tactic Name: volta_scudnn_128x32_relu_small_nn_v1 [05/23/2020-11:23:57] [V] [TRT] Tactic: -3946921629105938337 time 0.01536 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: -4420849921117327522 Time: 0.013312 [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaConvolution) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 time 0.01024 [05/23/2020-11:23:57] [V] [TRT] Tactic: 1 time 0.018432 [05/23/2020-11:23:57] [V] [TRT] Tactic: 2 time 0.014336 [05/23/2020-11:23:57] [V] [TRT] Tactic: 4 time 1.59642 [05/23/2020-11:23:57] [V] [TRT] Tactic: 5 time 0.033792 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0.01024 [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution] (CudaDepthwiseConvolution) [05/23/2020-11:23:57] [V] [TRT] CudaDepthwiseConvolution has no valid tactics for this config, skipping [05/23/2020-11:23:57] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: CudaConvolution Tactic: 0 [05/23/2020-11:23:57] [V] [TRT] [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800), Float(1,150,150,150) -> Float(1,150,150,4800) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:23:57] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:23:57] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,150,150:32,150), Float(1,150,150:32,150) -> Float(1,150,150:32,150) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 11) [ElementWise] (ElementWise) [05/23/2020-11:23:57] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:23:57] [V] [TRT] Tactic: 2 time 0.006144 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: decoder-input-of-previous-attention-output copy (Reformat) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 time 0.006144 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0.006144 [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Reformat) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 time 0.00624 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0.00624 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,150,150,4800) -> Float(1,32,4800) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 12) [Shuffle] (Shuffle) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,128,128), Float(1,128,128) -> Float(1,128,128), Float(1,128,128), Float(1,128,128) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [RNN] (RNNv2) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,32,4800), Float(1,64,2048) -> Float(1,64,9600) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 14) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,128,128), Float(1,64,8192) -> Float(1,64,64) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 16) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:23:57] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600), Float(1,64,9600), Float(1,64,64), Float(1,64,64) -> Float(1,64,9600) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]) (PointWise) [05/23/2020-11:23:57] [V] [TRT] Tactic: 128 time 0.008192 [05/23/2020-11:23:57] [V] [TRT] Tactic: 256 time 0.007168 [05/23/2020-11:23:57] [V] [TRT] Tactic: 512 time 0.008192 [05/23/2020-11:23:57] [V] [TRT] Tactic: -32 time 0.009216 [05/23/2020-11:23:57] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:23:57] [V] [TRT] Tactic: -128 time 0.008192 [05/23/2020-11:23:57] [V] [TRT] Fastest Tactic: 256 Time: 0.007168 [05/23/2020-11:23:57] [V] [TRT] *************** Autotuning format combination: Float(1,64,9600) -> Float(1,150) *************** [05/23/2020-11:23:57] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 22) [Reduce] (Reduce) [05/23/2020-11:23:57] [V] [TRT] Tactic: 1 time 0.009216 [05/23/2020-11:23:57] [V] [TRT] Tactic: 2 time 0.00624 [05/23/2020-11:23:57] [V] [TRT] Tactic: 3 time 0.01024 [05/23/2020-11:23:58] [V] [TRT] Tactic: 6 time 0.0512 [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 2 Time: 0.00624 [05/23/2020-11:23:58] [V] [TRT] *************** Autotuning format combination: Float(1,150) -> Float(1,150,150) *************** [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 23) [Shuffle] (Shuffle) [05/23/2020-11:23:58] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:58] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Int32(1,1,1) -> Float(1,150,150) *************** [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 24) [Ragged SoftMax] (RaggedSoftMax) [05/23/2020-11:23:58] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:58] [V] [TRT] *************** Autotuning format combination: Float(1,150,150) -> Float(1,1,150) *************** [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 29) [Shuffle] (Shuffle) [05/23/2020-11:23:58] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:58] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,150), Float(1,1,150) -> Float(1,1,150) *************** [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]) (PointWise) [05/23/2020-11:23:58] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:23:58] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:23:58] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:23:58] [V] [TRT] Tactic: -32 time 0.01024 [05/23/2020-11:23:58] [V] [TRT] Tactic: -64 time 0.008192 [05/23/2020-11:23:58] [V] [TRT] Tactic: -128 time 0.007168 [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:23:58] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,1,1) *************** [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 31) [Reduce] (Reduce) [05/23/2020-11:23:58] [V] [TRT] Tactic: 1 time 0.007168 [05/23/2020-11:23:58] [V] [TRT] Tactic: 3 time 0.011264 [05/23/2020-11:23:58] [V] [TRT] Tactic: 6 time 0.105472 [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 1 Time: 0.007168 [05/23/2020-11:23:58] [V] [TRT] *************** Autotuning format combination: Float(1,1,150), Float(1,1,1) -> Float(1,1,150) *************** [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 32) [ElementWise] (ElementWise) [05/23/2020-11:23:58] [V] [TRT] Tactic: 1 time 0.006144 [05/23/2020-11:23:58] [V] [TRT] Tactic: 2 time 0.007168 [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 1 Time: 0.006144 [05/23/2020-11:23:58] [V] [TRT] *************** Autotuning format combination: Float(1,1,150) -> Float(1,150,150) *************** [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 33) [Shuffle] (Shuffle) [05/23/2020-11:23:58] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:58] [V] [TRT] *************** Autotuning format combination: Float(1,150,150), Float(1,512,76800) -> Float(1,512,512) *************** [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 34) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:23:58] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:23:58] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Activation]_output copy (Reformat) [05/23/2020-11:23:58] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:23:58] [V] [TRT] *************** Autotuning format combination: Float(1,1152,1152), Float(1,256,512), Float(1,256,512) -> Float(1,256,256), Float(1,256,512), Float(1,256,512) *************** [05/23/2020-11:23:58] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN] (RNNv2) [05/23/2020-11:23:58] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:58] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:59] [V] [TRT] --------------- Timing Runner: decoder-output-attention copy (Reformat) [05/23/2020-11:23:59] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:23:59] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:23:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 36) [RNN]_output_1 copy (Reformat) [05/23/2020-11:23:59] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:23:59] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:23:59] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,30720) *************** [05/23/2020-11:23:59] [V] [TRT] *************** Autotuning format combination: Float(1,768,768), Float(1,40,30720) -> Float(1,40,40) *************** [05/23/2020-11:23:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 39) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:23:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:59] [V] [TRT] *************** Autotuning format combination: -> Float(1,40,40) *************** [05/23/2020-11:23:59] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,40,40) -> Float(1,40,40) *************** [05/23/2020-11:23:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation] (ElementWise) [05/23/2020-11:23:59] [V] [TRT] Tactic: 1 time 0.00512 [05/23/2020-11:23:59] [V] [TRT] Tactic: 2 time 0.00512 [05/23/2020-11:23:59] [V] [TRT] Fastest Tactic: 1 Time: 0.00512 [05/23/2020-11:23:59] [V] [TRT] *************** Autotuning format combination: -> Float(1,1,40) *************** [05/23/2020-11:23:59] [V] [TRT] *************** Autotuning format combination: Float(1,40,40), Float(1,1,40) -> Float(1,1,1) *************** [05/23/2020-11:23:59] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 44) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:23:59] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:23:59] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:23:59] [V] [TRT] *************** Autotuning format combination: Float(1,1,1) -> Float(1,1,1) *************** [05/23/2020-11:23:59] [V] [TRT] --------------- Timing Runner: PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]) (PointWise) [05/23/2020-11:23:59] [V] [TRT] Tactic: 128 time 0.00512 [05/23/2020-11:23:59] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:23:59] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:23:59] [V] [TRT] Fastest Tactic: 128 Time: 0.00512 [05/23/2020-11:23:59] [V] [TRT] Formats and tactics selection completed in 3.28116 seconds. [05/23/2020-11:23:59] [V] [TRT] After reformat layers: 42 layers [05/23/2020-11:23:59] [V] [TRT] Block size 1073741824 [05/23/2020-11:23:59] [V] [TRT] Block size 38400 [05/23/2020-11:23:59] [V] [TRT] Block size 38400 [05/23/2020-11:23:59] [V] [TRT] Block size 4608 [05/23/2020-11:23:59] [V] [TRT] Block size 2560 [05/23/2020-11:23:59] [V] [TRT] Block size 1024 [05/23/2020-11:23:59] [V] [TRT] Block size 1024 [05/23/2020-11:23:59] [V] [TRT] Block size 0 [05/23/2020-11:23:59] [V] [TRT] Total Activation Memory: 1073827840 [05/23/2020-11:23:59] [I] [TRT] Detected 11 inputs and 8 output network tensors. [05/23/2020-11:23:59] [V] [TRT] Engine generation completed in 3.63778 seconds. [05/23/2020-11:23:59] [V] [TRT] Engine Layer Information: [05/23/2020-11:23:59] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(40,640)] [05/23/2020-11:23:59] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,640)] [05/23/2020-11:23:59] [V] [TRT] Layer(Constant): (Unnamed Layer* 13) [Constant], Tactic: 0, -> (Unnamed Layer* 13) [Constant]_output[Float(32,64)] [05/23/2020-11:23:59] [V] [TRT] Layer(Constant): (Unnamed Layer* 15) [Constant], Tactic: 0, -> (Unnamed Layer* 15) [Constant]_output[Float(128,64)] [05/23/2020-11:23:59] [V] [TRT] Layer(Constant): (Unnamed Layer* 20) [Constant], Tactic: 0, -> (Unnamed Layer* 20) [Constant]_output[Float(1,64)] [05/23/2020-11:23:59] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 25) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:23:59] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 7) [Shuffle], Tactic: 0, decoder-input-of-previous-output-attention-alignment[Float(150,1)] -> (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:23:59] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 8) [Shuffle], Tactic: 0, input-conv_mask-for-every-sequence-length[Float(150)] -> (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] [05/23/2020-11:23:59] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 1) [Matrix Multiply], Tactic: 0, decoder-input-of-previous-output-frame[Float(1,40)], (Unnamed Layer* 0) [Constant]_output[Float(40,640)] -> (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)] [05/23/2020-11:23:59] [V] [TRT] Layer(Padding): (Unnamed Layer* 26) [Padding], Tactic: 0, (Unnamed Layer* 25) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 26) [Padding]_output[Float(1,1,150)] [05/23/2020-11:23:59] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 3) [ElementWise] + (Unnamed Layer* 4) [Activation], Tactic: 1, (Unnamed Layer* 1) [Matrix Multiply]_output[Float(1,640)], (Unnamed Layer* 2) [Constant]_output[Float(1,640)] -> (Unnamed Layer* 4) [Activation]_output[Float(1,640)] [05/23/2020-11:23:59] [V] [TRT] Layer(Convolution): (Unnamed Layer* 9) [Padding] + (Unnamed Layer* 10) [Convolution], Tactic: 0, (Unnamed Layer* 7) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)] [05/23/2020-11:23:59] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 11) [ElementWise], Tactic: 1, (Unnamed Layer* 10) [Convolution]_output[Float(32,1,150)], (Unnamed Layer* 8) [Shuffle]_output[Float(1,1,150)] -> (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] [05/23/2020-11:23:59] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,640)] [05/23/2020-11:23:59] [V] [TRT] Layer(Reformat): decoder-input-of-previous-attention-output copy, Tactic: 0, decoder-input-of-previous-attention-output[Float(1,512)] -> (Unnamed Layer* 5) [Concatenation]_output[Float(1,512)] [05/23/2020-11:23:59] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 12) [Shuffle], Tactic: 0, (Unnamed Layer* 11) [ElementWise]_output[Float(32,1,150)] -> (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)] [05/23/2020-11:23:59] [V] [TRT] Layer(RNN): (Unnamed Layer* 6) [RNN], Tactic: 0, (Unnamed Layer* 5) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-attention-hidden-state[Float(1,128)], decoder-input-of-previous-output-attention-cell-state[Float(1,128)] -> (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], decoder-output-attention-hidden-state[Float(1,128)], decoder-output-attention-cell-state[Float(1,128)] [05/23/2020-11:23:59] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 14) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 12) [Shuffle]_output[Float(150,32)], (Unnamed Layer* 13) [Constant]_output[Float(32,64)] -> (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)] [05/23/2020-11:23:59] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 16) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [RNN]_output_1[Float(1,128)], (Unnamed Layer* 15) [Constant]_output[Float(128,64)] -> (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)] [05/23/2020-11:23:59] [V] [TRT] Layer(PointWise): PWN(PWN(PWN((Unnamed Layer* 17) [ElementWise], (Unnamed Layer* 18) [ElementWise]), (Unnamed Layer* 19) [Activation]), (Unnamed Layer* 21) [ElementWise]), Tactic: 256, attention-keys[Float(150,64)], (Unnamed Layer* 14) [Matrix Multiply]_output[Float(150,64)], (Unnamed Layer* 16) [Matrix Multiply]_output[Float(1,64)], (Unnamed Layer* 20) [Constant]_output[Float(1,64)] -> (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] [05/23/2020-11:23:59] [V] [TRT] Layer(Reduce): (Unnamed Layer* 22) [Reduce], Tactic: 2, (Unnamed Layer* 21) [ElementWise]_output[Float(150,64)] -> (Unnamed Layer* 22) [Reduce]_output[Float(150)] [05/23/2020-11:23:59] [V] [TRT] Layer(RaggedSoftMax): (Unnamed Layer* 24) [Ragged SoftMax], Tactic: 0, (Unnamed Layer* 23) [Shuffle]_output[Float(1,150)], actual-encoder-input-sequence-length[Int32(1,1)] -> (Unnamed Layer* 24) [Ragged SoftMax]_output[Float(1,150)] [05/23/2020-11:23:59] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 28) [ElementWise], (Unnamed Layer* 30) [ElementWise]), Tactic: 128, decoder-input-of-previous-output-attention-alignment[Float(150,1)], (Unnamed Layer* 27) [Shuffle]_output[Float(150,1)], (Unnamed Layer* 29) [Shuffle]_output[Float(150,1)] -> (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] [05/23/2020-11:23:59] [V] [TRT] Layer(Reduce): (Unnamed Layer* 31) [Reduce], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)] -> (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] [05/23/2020-11:23:59] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 32) [ElementWise], Tactic: 1, (Unnamed Layer* 30) [ElementWise]_output[Float(150,1)], (Unnamed Layer* 31) [Reduce]_output[Float(1,1)] -> (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] [05/23/2020-11:23:59] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 33) [Shuffle], Tactic: 0, (Unnamed Layer* 32) [ElementWise]_output[Float(150,1)] -> decoder-output-alignment[Float(1,150)] [05/23/2020-11:23:59] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 34) [Matrix Multiply], Tactic: 0, decoder-output-alignment[Float(1,150)], encoder-output-cat-embedding-data[Float(150,512)] -> decoder-output-attention[Float(1,512)] [05/23/2020-11:23:59] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,512)] [05/23/2020-11:23:59] [V] [TRT] Layer(Reformat): (Unnamed Layer* 4) [Activation]_output copy, Tactic: 0, (Unnamed Layer* 4) [Activation]_output[Float(1,640)] -> (Unnamed Layer* 35) [Concatenation]_output[Float(1,640)] [05/23/2020-11:23:59] [V] [TRT] Layer(RNN): (Unnamed Layer* 36) [RNN], Tactic: 0, (Unnamed Layer* 35) [Concatenation]_output[Float(1,1152)], decoder-input-of-previous-output-lstm-hidden-state[Float(2,256)], decoder-input-of-previous-output-lstm-cell-state[Float(2,256)] -> (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)], decoder-output-lstm-hidden-state[Float(2,256)], decoder-output-lstm-cell-state[Float(2,256)] [05/23/2020-11:23:59] [V] [TRT] Layer(Reformat): decoder-output-attention copy, Tactic: 0, decoder-output-attention[Float(1,512)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,512)] [05/23/2020-11:23:59] [V] [TRT] Layer(Reformat): (Unnamed Layer* 36) [RNN]_output_1 copy, Tactic: 0, (Unnamed Layer* 36) [RNN]_output_1[Float(1,256)] -> (Unnamed Layer* 37) [Concatenation]_output[Float(1,256)] [05/23/2020-11:23:59] [V] [TRT] Layer(Constant): (Unnamed Layer* 38) [Constant], Tactic: 0, -> (Unnamed Layer* 38) [Constant]_output[Float(768,40)] [05/23/2020-11:23:59] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 39) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 37) [Concatenation]_output[Float(1,768)], (Unnamed Layer* 38) [Constant]_output[Float(768,40)] -> (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)] [05/23/2020-11:23:59] [V] [TRT] Layer(Constant): (Unnamed Layer* 40) [Constant], Tactic: 0, -> (Unnamed Layer* 40) [Constant]_output[Float(1,40)] [05/23/2020-11:23:59] [V] [TRT] Layer(ElementWise): (Unnamed Layer* 41) [ElementWise] + (Unnamed Layer* 42) [Activation], Tactic: 1, (Unnamed Layer* 39) [Matrix Multiply]_output[Float(1,40)], (Unnamed Layer* 40) [Constant]_output[Float(1,40)] -> decoder-ouput-frame[Float(1,40)] [05/23/2020-11:23:59] [V] [TRT] Layer(Constant): (Unnamed Layer* 43) [Constant], Tactic: 0, -> (Unnamed Layer* 43) [Constant]_output[Float(40,1)] [05/23/2020-11:23:59] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 44) [Matrix Multiply], Tactic: 0, decoder-ouput-frame[Float(1,40)], (Unnamed Layer* 43) [Constant]_output[Float(40,1)] -> (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] [05/23/2020-11:23:59] [V] [TRT] Layer(PointWise): PWN(PWN((Unnamed Layer* 45) [Constant], (Unnamed Layer* 46) [ElementWise]), (Unnamed Layer* 47) [Activation]), Tactic: 128, (Unnamed Layer* 44) [Matrix Multiply]_output[Float(1,1)] -> stop-token[Float(1,1)] [05/23/2020-11:23:59] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:23:59] [V] [TRT] Original: 12 layers [05/23/2020-11:23:59] [V] [TRT] After dead-layer removal: 12 layers [05/23/2020-11:23:59] [V] [TRT] After Myelin optimization: 12 layers [05/23/2020-11:23:59] [V] [TRT] After scale fusion: 12 layers [05/23/2020-11:23:59] [V] [TRT] Fusing (Unnamed Layer* 5) [ElementWise] with (Unnamed Layer* 6) [ElementWise] [05/23/2020-11:23:59] [V] [TRT] Fusing (Unnamed Layer* 9) [ElementWise] with (Unnamed Layer* 10) [ElementWise] [05/23/2020-11:23:59] [V] [TRT] After vertical fusions: 10 layers [05/23/2020-11:23:59] [V] [TRT] After final dead-layer removal: 10 layers [05/23/2020-11:23:59] [V] [TRT] After tensor merging: 10 layers [05/23/2020-11:23:59] [V] [TRT] Eliminating concatenation (Unnamed Layer* 11) [Concatenation] [05/23/2020-11:23:59] [V] [TRT] Generating copy for (Unnamed Layer* 6) [ElementWise]_output to rout-output [05/23/2020-11:23:59] [V] [TRT] Generating copy for rout-output-hidden-state to rout-output [05/23/2020-11:23:59] [V] [TRT] After concat removal: 11 layers [05/23/2020-11:23:59] [V] [TRT] Graph construction and optimization completed in 0.00609927 seconds. [05/23/2020-11:24:00] [V] [TRT] Constructing optimization profile number 0 out of 1 *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:24:00] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,400) *************** [05/23/2020-11:24:00] [V] [TRT] *************** Autotuning format combination: -> Float(1,20,20) *************** [05/23/2020-11:24:00] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:24:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 3) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:24:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:24:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:24:00] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:24:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 4) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:24:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:24:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:24:00] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:24:00] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]) (PointWise) [05/23/2020-11:24:00] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:24:00] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:24:00] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:24:00] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:24:00] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:24:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 7) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:24:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:24:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:24:00] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,400) -> Float(1,20,20) *************** [05/23/2020-11:24:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 8) [Matrix Multiply] (MatrixMultiply) [05/23/2020-11:24:00] [V] [TRT] Tactic: 0 is the only option, timing skipped [05/23/2020-11:24:00] [V] [TRT] Fastest Tactic: 0 Time: 0 [05/23/2020-11:24:00] [V] [TRT] *************** Autotuning format combination: Float(1,20,20), Float(1,20,20), Float(1,20,20) -> Float(1,20,20) *************** [05/23/2020-11:24:00] [V] [TRT] --------------- Timing Runner: PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]) (PointWise) [05/23/2020-11:24:00] [V] [TRT] Tactic: 128 time 0.006144 [05/23/2020-11:24:00] [V] [TRT] Tactic: 256 time 0.006144 [05/23/2020-11:24:00] [V] [TRT] Tactic: 512 time 0.006144 [05/23/2020-11:24:00] [V] [TRT] Fastest Tactic: 128 Time: 0.006144 [05/23/2020-11:24:00] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 6) [ElementWise]_output copy (Reformat) [05/23/2020-11:24:00] [V] [TRT] Tactic: 0 time 0.00512 [05/23/2020-11:24:00] [V] [TRT] Fastest Tactic: 0 Time: 0.00512 [05/23/2020-11:24:01] [V] [TRT] --------------- Timing Runner: rout-output-hidden-state copy (Reformat) [05/23/2020-11:24:01] [V] [TRT] Tactic: 0 time 0.005184 [05/23/2020-11:24:01] [V] [TRT] Fastest Tactic: 0 Time: 0.005184 [05/23/2020-11:24:01] [V] [TRT] Formats and tactics selection completed in 1.04373 seconds. [05/23/2020-11:24:01] [V] [TRT] After reformat layers: 11 layers [05/23/2020-11:24:01] [V] [TRT] Block size 1073741824 [05/23/2020-11:24:01] [V] [TRT] Block size 512 [05/23/2020-11:24:01] [V] [TRT] Block size 512 [05/23/2020-11:24:01] [V] [TRT] Block size 512 [05/23/2020-11:24:01] [V] [TRT] Total Activation Memory: 1073743360 [05/23/2020-11:24:01] [I] [TRT] Detected 3 inputs and 4 output network tensors. [05/23/2020-11:24:01] [V] [TRT] Engine generation completed in 1.26989 seconds. [05/23/2020-11:24:01] [V] [TRT] Engine Layer Information: [05/23/2020-11:24:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 0) [Constant], Tactic: 0, -> (Unnamed Layer* 0) [Constant]_output[Float(20,20)] [05/23/2020-11:24:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 1) [Constant], Tactic: 0, -> (Unnamed Layer* 1) [Constant]_output[Float(20,20)] [05/23/2020-11:24:01] [V] [TRT] Layer(Constant): (Unnamed Layer* 2) [Constant], Tactic: 0, -> (Unnamed Layer* 2) [Constant]_output[Float(1,20)] [05/23/2020-11:24:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 3) [Matrix Multiply], Tactic: 0, rout-input0[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:24:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 4) [Matrix Multiply], Tactic: 0, rout-input-of-previous-output-rout-hidden-state[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:24:01] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 5) [ElementWise], (Unnamed Layer* 6) [ElementWise]), Tactic: 128, (Unnamed Layer* 4) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 3) [Matrix Multiply]_output[Float(1,20)] -> (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] [05/23/2020-11:24:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 7) [Matrix Multiply], Tactic: 0, rout-input1[Float(1,20)], (Unnamed Layer* 0) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:24:01] [V] [TRT] Layer(MatrixMultiply): (Unnamed Layer* 8) [Matrix Multiply], Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)], (Unnamed Layer* 1) [Constant]_output[Float(20,20)] -> (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)] [05/23/2020-11:24:01] [V] [TRT] Layer(PointWise): PWN((Unnamed Layer* 9) [ElementWise], (Unnamed Layer* 10) [ElementWise]), Tactic: 128, (Unnamed Layer* 8) [Matrix Multiply]_output[Float(1,20)], (Unnamed Layer* 2) [Constant]_output[Float(1,20)], (Unnamed Layer* 7) [Matrix Multiply]_output[Float(1,20)] -> rout-output-hidden-state[Float(1,20)] [05/23/2020-11:24:01] [V] [TRT] Layer(Reformat): (Unnamed Layer* 6) [ElementWise]_output copy, Tactic: 0, (Unnamed Layer* 6) [ElementWise]_output[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:24:01] [V] [TRT] Layer(Reformat): rout-output-hidden-state copy, Tactic: 0, rout-output-hidden-state[Float(1,20)] -> rout-output[Float(1,20)] [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:01] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:02] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:03] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:04] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles [05/23/2020-11:24:05] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles thread0 load float count:3834 thread1 load float count:3834 thread3 load float count:3834 thread6 load float count:3834 thread7 load float count:3834 thread9 load float count:3834 thread19 load float count:3834 thread18 load float count:3834 thread4 load float count:3834 thread2 load float count:3834 thread11 load float count:3834 thread13 load float count:3834 thread15 load float count:3834 thread12 load float count:3834 thread10 load float count:3834 thread14 load float count:3834 thread16 load float count:3834 thread17 load float count:3834 thread8 load float count:3834 thread5 load float count:3834 stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 4 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 0 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 19 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 15 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 9 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 18 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 13 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 6 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 2 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 3 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 12 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 7 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 14 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 5 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 11 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 10 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 1 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 17 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 16 finish stop token triggered at step: 327, batch_id: 0, 0.999942 The output sequence length is 654 thread 8 finish finish tacotron release called destructor called Summary: ver=2, add following hparam fields: (1) need_denorm Header: magic: 'TTS' (3 bytes) ver : 2 (1 byte) header_size: 20 (4 bytes) hparam_count: 20 (4 bytes) weight_count: 20 (4 bytes) norm_count: 40 (4 bytes) HPARMAS: model_config->mechanism:1 model_config->OutLengthTimesInLength:34 model_config->FramesOneStep:2 model_config->encoder_input_channels:71 model_config->encoder_conv_layers:2 model_config->encoder_conv_width:5 model_config->encoder_conv_channels:256 model_config->encoder_lstm_layers:1 model_config->encoder_lstm_channels:512 model_config->decoder_pre_layers:1 model_config->decoder_pre_channels:640 model_config->decoder_attention_channels:64 model_config->decoder_attention_lstm_channels:128 model_config->decoder_attention_conv_width:31 model_config->decoder_attention_conv_channels:32 model_config->decoder_lstm_layers:2 model_config->decoder_lstm_channels:256 model_config->decoder_output_channels:40 (1+)model_config->encoder_voiceprint_embedding_channels:0 (2+)model_config->need_denorm:1 [05/23/2020-11:24:45] [V] [TRT] Applying generic optimizations to the graph for inference. [05/23/2020-11:24:45] [V] [TRT] Original: 18 layers [05/23/2020-11:24:45] [V] [TRT] After dead-layer removal: 18 layers [05/23/2020-11:24:45] [V] [TRT] After Myelin optimization: 18 layers [05/23/2020-11:24:45] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 3) [Convolution] with scale (Unnamed Layer* 4) [Scale] [05/23/2020-11:24:45] [V] [TRT] Fusing convolution weights from (Unnamed Layer* 8) [Convolution] with scale (Unnamed Layer* 9) [Scale] [05/23/2020-11:24:45] [V] [TRT] After scale fusion: 16 layers [05/23/2020-11:24:45] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:24:45] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:24:45] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:24:45] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:24:45] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] with (Unnamed Layer* 3) [Convolution] [05/23/2020-11:24:45] [V] [TRT] Fusing (Unnamed Layer* 2) [Padding] + (Unnamed Layer* 3) [Convolution] with (Unnamed Layer* 5) [Activation] [05/23/2020-11:24:45] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:24:45] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:24:45] [W] [TRT] Can't fuse pad and convolution with same pad mode [05/23/2020-11:24:45] [W] [TRT] Can't fuse pad and convolution with caffe pad mode [05/23/2020-11:24:45] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] with (Unnamed Layer* 8) [Convolution] [05/23/2020-11:24:45] [V] [TRT] Fusing (Unnamed Layer* 7) [Padding] + (Unnamed Layer* 8) [Convolution] with (Unnamed Layer* 10) [Activation] [05/23/2020-11:24:45] [V] [TRT] After vertical fusions: 12 layers [05/23/2020-11:24:45] [V] [TRT] After final dead-layer removal: 12 layers [05/23/2020-11:24:45] [V] [TRT] After tensor merging: 12 layers [05/23/2020-11:24:45] [V] [TRT] After concat removal: 12 layers [05/23/2020-11:24:45] [V] [TRT] Graph construction and optimization completed in 0.0713237 seconds.