TF-TRT error in calibrate()

Environment: ubuntu 16.04, nvidia container 19.09-py3 nvcr.io/nvidia/tensorflow

I want to using TF-TRT INT8 quantization. so I did follow your examples with my network.
But I blocked by some error.
The error occured in follow function.

Please help me.

Code:
trt_graph = converter.calibrate(
fetch_names=[‘detection_boxes:0’, ‘detection_scores:0’, ‘detection_classes:0’, ‘num_detections:0’, ‘detection_keypoints:0’],
num_runs=10,
feed_dict_fn=lambda: {‘image_tensor:0’: data}) # There are 10 images in data

Logs:
2019-10-31 07:03:37.304493: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:716] Starting calibration thread on device 0, Calibration Resource @ 0x7f5cac001880
2019-10-31 07:03:37.304881: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
2019-10-31 07:03:37.322659: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:716] Starting calibration thread on device 0, Calibration Resource @ 0x7f5cac0093b0
2019-10-31 07:03:37.322950: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:716] Starting calibration thread on device 0, Calibration Resource @ 0x7f5cac0090d0
2019-10-31 07:03:37.323076: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
2019-10-31 07:03:37.323392: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
2019-10-31 07:03:37.336244: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:716] Starting calibration thread on device 0, Calibration Resource @ 0x7f5cbc55cf50
2019-10-31 07:03:37.336843: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
2019-10-31 07:03:37.348520: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:716] Starting calibration thread on device 0, Calibration Resource @ 0x7f5cb80021b0
2019-10-31 07:03:37.349075: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
2019-10-31 07:06:01.800437: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at get_serialized_resource_op.cc:48 : Not found: Resource TF-TRT-Calibration/TRTEngineOp_6/N10tensorflow8tensorrt24SerializableResourceBaseE does not exist.
Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1356, in _do_call
return fn(*args)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
(0) Not found: Resource TF-TRT-Calibration/TRTEngineOp_6/N10tensorflow8tensorrt24SerializableResourceBaseE does not exist.
[[{{node GetSerializedResourceOp_1}}]]
(1) Not found: Resource TF-TRT-Calibration/TRTEngineOp_6/N10tensorflow8tensorrt24SerializableResourceBaseE does not exist.
[[{{node GetSerializedResourceOp_1}}]]
[[GetSerializedResourceOp_1/_175]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/root/.vscode-server/extensions/ms-python.python-2019.10.41019/pythonFiles/ptvsd_launcher.py”, line 43, in
main(ptvsdArgs)
File “/root/.vscode-server/extensions/ms-python.python-2019.10.41019/pythonFiles/lib/python/old_ptvsd/ptvsd/main.py”, line 432, in main
run()
File “/root/.vscode-server/extensions/ms-python.python-2019.10.41019/pythonFiles/lib/python/old_ptvsd/ptvsd/main.py”, line 316, in run_file
runpy.run_path(target, run_name=‘main’)
File “/usr/lib/python3.6/runpy.py”, line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File “/usr/lib/python3.6/runpy.py”, line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File “/usr/lib/python3.6/runpy.py”, line 85, in _run_code
exec(code, run_globals)
File “/workspace/Projects/TensorRT_Practice/test1.py”, line 61, in
feed_dict_fn=lambda: {‘image_tensor:0’: data})
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/compiler/tensorrt/trt_convert.py”, line 355, in calibrate
self.finalize_calibration()
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/compiler/tensorrt/trt_convert.py”, line 813, in finalize_calibration
resource_name_input: node.name
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 950, in run
run_metadata_ptr)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1173, in _run
feed_dict_tensor, options, run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1350, in _do_run
run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
(0) Not found: Resource TF-TRT-Calibration/TRTEngineOp_6/N10tensorflow8tensorrt24SerializableResourceBaseE does not exist.
[[node GetSerializedResourceOp_1 (defined at usr/local/lib/python3.6/dist-packages/tensorflow/compiler/tf2tensorrt/ops/gen_trt_ops.py:285) ]]
(1) Not found: Resource TF-TRT-Calibration/TRTEngineOp_6/N10tensorflow8tensorrt24SerializableResourceBaseE does not exist.
[[node GetSerializedResourceOp_1 (defined at usr/local/lib/python3.6/dist-packages/tensorflow/compiler/tf2tensorrt/ops/gen_trt_ops.py:285) ]]
[[GetSerializedResourceOp_1/_175]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node GetSerializedResourceOp_1:
Placeholder_1 (defined at workspace/Projects/TensorRT_Practice/test1.py:61)

Input Source operations connected to node GetSerializedResourceOp_1:
Placeholder_1 (defined at workspace/Projects/TensorRT_Practice/test1.py:61)

Original stack trace for ‘GetSerializedResourceOp_1’:
File “root/.vscode-server/extensions/ms-python.python-2019.10.41019/pythonFiles/ptvsd_launcher.py”, line 43, in
main(ptvsdArgs)
File “root/.vscode-server/extensions/ms-python.python-2019.10.41019/pythonFiles/lib/python/old_ptvsd/ptvsd/main.py”, line 432, in main
run()
File “root/.vscode-server/extensions/ms-python.python-2019.10.41019/pythonFiles/lib/python/old_ptvsd/ptvsd/main.py”, line 316, in run_file
runpy.run_path(target, run_name=‘main’)
File “usr/lib/python3.6/runpy.py”, line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File “usr/lib/python3.6/runpy.py”, line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File “usr/lib/python3.6/runpy.py”, line 85, in _run_code
exec(code, run_globals)
File “workspace/Projects/TensorRT_Practice/test1.py”, line 61, in
feed_dict_fn=lambda: {‘image_tensor:0’: data})
File “usr/local/lib/python3.6/dist-packages/tensorflow/python/compiler/tensorrt/trt_convert.py”, line 355, in calibrate
self.finalize_calibration()
File “usr/local/lib/python3.6/dist-packages/tensorflow/python/compiler/tensorrt/trt_convert.py”, line 804, in finalize_calibration
container_input, resource_name_input))
File “usr/local/lib/python3.6/dist-packages/tensorflow/compiler/tf2tensorrt/ops/gen_trt_ops.py”, line 285, in get_serialized_resource_op
resource_name=resource_name, name=name)
File “usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py”, line 788, in _apply_op_helper
op_def=op_def)
File “usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File “usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py”, line 3616, in create_op
op_def=op_def)
File “usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py”, line 2005, in init
self._traceback = tf_stack.extract_stack()

This could be because converter.convert() is not called before converter.calibrate()?

@Pooya-Davoodi

No, I already called convert() before calibrate() in entire source code.
Thanks.

Could you share the complete source code, TF-TRT verbose log, and the input model?

@Pooya-Davoodi

Thank you for helping me.
This is my code, network(.pb) and images for calibration.
https://drive.google.com/open?id=1J7wq9TY3j3-kTmaWqePoq7-SWHjCBykd

did you set session_config parameter in TrtGraphConverter()?
I set gpu per_process_gpu_memery_fraction=0.4, everything is ok.