Value for attr 'OutT' of int32 is not in the list of allowed values: int8, half, float

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I optimized an NLP model using the approach suggested here: https://devtalk.nvidia.com/default/topic/1050151/tensorrt/float_ref-error-when-using-tf-trt/

When I run the model in serving, hitting it with a query triggers this error:

2019-04-18 19:52:43.992028: E external/org_tensorflow/tensorflow/core/common_runtime/executor.cc:624] Executor failed to create kernel. Invalid argument: Value for attr 'OutT' of int32 is not in the list of allowed values: int8, half, float
        ; NodeDef: {{node import/multi_transformer/TRTEngineOp_0}}; Op<name=TRTEngineOp; signature=in_tensor: -> out_tensor:; attr=serialized_segment:string; attr=input_shapes:list(shape); attr=output_shapes:list(shape); attr=segment_funcdef_name:string; attr=InT:list(type),min=1,allowed=[DT_INT8, DT_HALF, DT_FLOAT]; attr=OutT:list(type),min=1,allowed=[DT_INT8, DT_HALF, DT_FLOAT]; attr=static_engine:bool,default=true; attr=fixed_input_size:bool,default=true; attr=cached_engine_batches:list(int),default=[]; attr=max_cached_engines_count:int,default=1; attr=workspace_size_bytes:int; attr=precision_mode:string,allowed=["FP32", "FP16", "INT8"]; attr=calibration_data:string,default=""; attr=use_calibration:bool,default=true>
         [[{{node import/multi_transformer/TRTEngineOp_0}}]]
2

The model is modified from the Transformer model from Tensor2Tensor, if that is relevant.

3

Verified that the same error occurs when optimizing the built-in Transformer model from T2T on the built-in translate_ende_wmt8k problem.

Steps to reproduce:

  • Download and set up tensor2tensor
  • python ~/tensor2tensor/tensor2tensor/bin/t2t-datagen --problem=translate_ende_wmt8k --data_dir=input
  • python ~/tensor2tensor/tensor2tensor/bin/t2t-trainer --problem=translate_ende_wmt8k --model=transformer --hparams_set=transformer_base_single_gpu --data_dir=input --output_dir=output --train_steps=2500000
  • python ~/tensor2tensor/tensor2tensor/bin/t2t-exporter --model transformer --output_dir=output --data_dir=input --hparams_set=transformer_base_single_gpu --problem translate_ende_wmt8k
  • Optimize the exported model with TF-TRT (required some trickery with replacing RefEnter nodes with Enter nodes for me)
  • Serve the resulting model using docker and this image: Docker Hub
4

Oh, and for reproduction purposes, don’t need to fully optimize the model; 1000 steps of training should be fine.

5

I have the same problem

tensorflow.python.framework.errors_impl.InvalidArgumentError: Value for attr 'OutT' of int32 is not in the list of allowed values: int8, half, float
	; NodeDef: {{node decoder/while/TRTEngineOp_30}}; Op<name=TRTEngineOp; signature=in_tensor: -> out_tensor:; attr=serialized_segment:string; attr=input_shapes:list(shape); attr=output_shapes:list(shape); attr=segment_funcdef_name:string; attr=InT:list(type),min=1,allowed=[DT_INT8, DT_HALF, DT_FLOAT]; attr=OutT:list(type),min=1,allowed=[DT_INT8, DT_HALF, DT_FLOAT]; attr=static_engine:bool,default=true; attr=fixed_input_size:bool,default=true; attr=cached_engine_batches:list(int),default=[]; attr=max_cached_engines_count:int,default=1; attr=workspace_size_bytes:int; attr=precision_mode:string,allowed=["FP32", "FP16", "INT8"]; attr=calibration_data:string,default=""; attr=use_calibration:bool,default=true>