I am trying to convert my Tensorflow model to TRT following the object detection sample.
My command is
python object_detection.py --input_saved_model_dir /workspace/examples/NumPlateDetection/saved_model --output_saved_model_dir /workspace/examples/NumPlateDetection --data_dir /workspace/examples/NumPlateDetection/infer/images --calib_data_dir /workspace/examples/NumPlateDetection/images --optimize_offline --precision INT8 --num_calib_inputs 800 --input_size 736 --batch_size 8 --mode 'inference' --outputimg_path /workspace/examples/NumPlateDetection/outputs --use_trt
What could be issue?
I have the following errors
: 164 curr_region_allocation_bytes_: 34359738368
2020-06-17 03:38:17.241209: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats:
Limit: 23469584548
InUse: 17155265792
MaxInUse: 19082024704
NumAllocs: 4030
MaxAllocSize: 3833987072
2020-06-17 03:38:17.241298: W tensorflow/core/common_runtime/bfc_allocator.cc:429] *********_______________******************_***********_______***************************************
2020-06-17 03:38:17.241333: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger Requested amount of GPU memory (4404019200 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
2020-06-17 03:38:17.241362: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger /home/jenkins/workspace/TensorRT/helpers/rel-7.0/L1_Nightly/build/source/rtSafe/resources.h (164) - OutOfMemory Error in GpuMemory: 0
2020-06-17 03:38:17.241455: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger Out of memory error during getBestTactic: (Unnamed Layer* 0) [Shuffle]
2020-06-17 03:38:17.241481: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger Internal error: could not find any implementation for node (Unnamed Layer* 0) [Shuffle], try increasing the workspace size with IBuilder::setMaxWorkspaceSize()
2020-06-17 03:38:17.243854: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../builder/tacticOptimizer.cpp (1523) - OutOfMemory Error in computeCosts: 0
2020-06-17 03:38:17.255722: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:841] Calibration failed: Internal: Failed to build TensorRT engine
2020-06-17 03:38:17.255955: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Internal: Failed to feed calibration data
[[{{node TRTEngineOp_31}}]]
[[SecondStagePostprocessor/map/while/Switch_1/_316]]
2020-06-17 03:38:17.256282: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Internal: Failed to feed calibration data
[[{{node TRTEngineOp_31}}]]
Traceback (most recent call last):
File "numplate_detection.py", line 380, in <module>
optimize_offline=args.optimize_offline)
File "numplate_detection.py", line 107, in get_graph_func
input_fn, calib_data_dir, num_calib_inputs//batch_size))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/compiler/tensorrt/trt_convert.py", line 1004, in convert
self._converted_func(*map(ops.convert_to_tensor, inp))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1551, in __call__
return self._call_impl(args, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1591, in _call_impl
return self._call_flat(args, self.captured_inputs, cancellation_manager)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1692, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 545, in call
ctx=ctx)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Failed to feed calibration data
[[node TRTEngineOp_31 (defined at numplate_detection.py:107) ]]
[[SecondStagePostprocessor/map/while/Switch_1/_316]]
(1) Internal: Failed to feed calibration data
[[node TRTEngineOp_31 (defined at numplate_detection.py:107) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_pruned_27865]
Function call stack:
pruned -> pruned
terminate called without an active exception
Aborted (core dumped)