I am encountering two different errors while trying to convert a saved model, derived from Google’s BERT NLP model, to a TensorRT model/engine.
I am using instructions I found here: Optimizing TensorFlow Serving performance with NVIDIA TensorRT | by TensorFlow | TensorFlow | Medium
The first error occurs when attempting the conversion on a Linux server configured with 1 T4 GPU:
2019-06-25 19:51:26.566401: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] TensorRTOptimizer: Invalid argument: Input 0 of node module_apply_tokens/bert/embeddings/embedding_lookup was passed float from module/bert/embeddings/word_embeddings:0 incompatible with expected resource.
The conversion appears successful on a Linux server with 4 T4 GPU’s installed:
2019-06-25 19:56:48.679553: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: tf_graph
2019-06-25 19:56:48.679644: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 1433 nodes (-200), 1728 edges (-202), time = 2040.073ms.
2019-06-25 19:56:48.679669: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 1433 nodes (0), 1728 edges (0), time = 576.801ms.
2019-06-25 19:56:48.679690: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 1433 nodes (0), 1728 edges (0), time = 998.658ms.
But, when the resulting converted model is loaded at runtime, the following error occurs:
2019-06-25 20:00:09.511052: E tensorflow_serving/util/retrier.cc:37] Loading servable: {name: bert_bot_model version: 1559237901} failed: Invalid argument: Tensor module/vocab_file:0, specified in either feed_devices or fetch_devices was not found in the Graph
The following command was used to load/serve the TensorRT model:
docker run --rm --runtime=nvidia -p 9001:8500 -p 8001:8501 --name tensorflow_gpu -v /models/bert_bot_model_fp32:/models/bert_bot_model -e MODEL_NAME=bert_bot_model -t tensorflow/serving:latest-gpu &
Any help in resolving one or both of these issues would be very much appreciated.