Table not initialized with TF-TRT

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I want to convert my model to TRT-INT8, however, when i get into converter.calibrate function, i get the error:
File “/usr/lib64/python2.7/site-packages/tensorflow_core/python/compiler/tensorrt/trt_convert.py”, line 612, in calibrate
fetches, feed_dict=feed_dict_fn() if feed_dict_fn else None)
File “/usr/lib64/python2.7/site-packages/tensorflow_core/python/client/session.py”, line 956, in run
run_metadata_ptr)
File “/usr/lib64/python2.7/site-packages/tensorflow_core/python/client/session.py”, line 1180, in _run
feed_dict_tensor, options, run_metadata)
File “/usr/lib64/python2.7/site-packages/tensorflow_core/python/client/session.py”, line 1359, in _do_run
run_metadata)
File “/usr/lib64/python2.7/site-packages/tensorflow_core/python/client/session.py”, line 1384, in _do_call
raise type(e)(node_def, op, message)
FailedPreconditionError: Table not initialized.
[[node index_to_string_Lookup (defined at usr/lib64/python2.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Node index_to_string_Lookup is a output node of my model.

I googled this problem and get the an suggest to add sess.run(tf.tables_initializer()), so i change the calibrate func by add calibration_sess.run(tf.tables_initializer()):

with session.Session(
  graph=self._calibration_graph,
  config=self._session_config) as calibration_sess:
  calibration_sess.run(tf.tables_initializer())
   for _ in range(num_runs):
     calibration_sess.run(
        fetches, feed_dict=feed_dict_fn() if feed_dict_fn else None)

but i failed, how can i slove this problem?

2

Hi,

Could you please share the below platform details so we can help better?
o Linux distro and version
o GPU type
o Nvidia driver version
o CUDA version
o CUDNN version
o Python version [if using python]
o Tensorflow and PyTorch version
o TensorRT version

Also, if possible please share the error log and repro script.

Thanks

3

HI,
Thankyou for your response, i sloved this problem by add calibration_sess.run(self._calibration_graph.get_operation_by_name(‘legacy_init_op’)) other than calibration_sess.run(tf.tables_initializer()) in above code.
This op in my model should run fisrt to do initializer work, i wonder weather TF-TRT provide any interface so that i don’t need to change the TF-TRT code?

4

Hi,

This seems to be more like a TF function. I don’t think TF-TRT provide any such interface.

Thanks