Hi Guys,
I have successfully trained an object detection model using TLT. When I evaluate the saved model, I get the following error.
Using TensorFlow backend.
2019-10-30 09:55:21,449 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/examples/detectnet_v2_resnet10/specs/detectnet_v2_train_resnet10_kitti.txt
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-10-30 09:55:21,764 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-10-30 09:55:22.583449: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-30 09:55:23.131657: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x7ee4a70 executing computations on platform CUDA. Devices:
2019-10-30 09:55:23.131710: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-10-30 09:55:23.131720: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-10-30 09:55:23.131728: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-10-30 09:55:23.131737: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-10-30 09:55:23.131744: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (4): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-10-30 09:55:23.160988: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3497905000 Hz
2019-10-30 09:55:23.161797: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x804d500 executing computations on platform Host. Devices:
2019-10-30 09:55:23.161829: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-30 09:55:23.161978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:05:00.0
totalMemory: 10.76GiB freeMemory: 10.29GiB
2019-10-30 09:55:23.162040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 1 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:06:00.0
totalMemory: 10.76GiB freeMemory: 9.75GiB
2019-10-30 09:55:23.162093: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 2 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:07:00.0
totalMemory: 10.76GiB freeMemory: 9.74GiB
2019-10-30 09:55:23.162146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 3 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:0a:00.0
totalMemory: 10.76GiB freeMemory: 9.75GiB
2019-10-30 09:55:23.162198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 4 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:0b:00.0
totalMemory: 10.76GiB freeMemory: 9.75GiB
2019-10-30 09:55:23.162963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1, 2, 3, 4
2019-10-30 09:55:23.168607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-30 09:55:23.168627: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 1 2 3 4
2019-10-30 09:55:23.168635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N N N N N
2019-10-30 09:55:23.168642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N N N N N
2019-10-30 09:55:23.168648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N N N N N
2019-10-30 09:55:23.168655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N N N N N
2019-10-30 09:55:23.168663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 4: N N N N N
2019-10-30 09:55:23.168806: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10013 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:05:00.0, compute capability: 7.5)
2019-10-30 09:55:23.169044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9483 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:06:00.0, compute capability: 7.5)
2019-10-30 09:55:23.169261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9472 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:07:00.0, compute capability: 7.5)
2019-10-30 09:55:23.169542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 9485 MB memory) -> physical GPU (device: 3, name: GeForce RTX 2080 Ti, pci bus id: 0000:0a:00.0, compute capability: 7.5)
2019-10-30 09:55:23.169790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 9485 MB memory) -> physical GPU (device: 4, name: GeForce RTX 2080 Ti, pci bus id: 0000:0b:00.0, compute capability: 7.5)
/usr/local/lib/python2.7/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
warnings.warn('No training configuration found in save file: '
WARNING:tensorflow:From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
2019-10-30 09:55:24,957 [WARNING] tensorflow: From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
2019-10-30 09:55:32,554 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/evaluation/build_evaluator.pyc: Found 16990 samples in validation set
Traceback (most recent call last):
File "/usr/local/bin/tlt-evaluate", line 10, in <module>
sys.exit(main())
File "./common/magnet_evaluate.py", line 38, in main
File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-2>", line 2, in main
File "./detectnet_v2/utilities/timer.py", line 46, in wrapped_fn
File "./detectnet_v2/scripts/evaluate.py", line 119, in main
File "./detectnet_v2/evaluation/build_evaluator.py", line 124, in build_evaluator_for_trained_gridbox
File "./detectnet_v2/model/utilities.py", line 26, in _fn_wrapper
File "./detectnet_v2/model/detectnet_model.py", line 617, in build_validation_graph
File "./detectnet_v2/model/utilities.py", line 26, in _fn_wrapper
File "./detectnet_v2/model/detectnet_model.py", line 582, in build_inference_graph
File "./detectnet_v2/model/detectnet_model.py", line 243, in predictions_to_dict
File "./detectnet_v2/objectives/base_objective.py", line 97, in reshape_output
File "/usr/local/lib/python2.7/dist-packages/keras/engine/base_layer.py", line 457, in __call__
output = self.call(inputs, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/layers/core.py", line 401, in call
return K.reshape(inputs, (K.shape(inputs)[0],) + self.target_shape)
File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 1969, in reshape
return tf.reshape(x, shape)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 7179, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1823, in __init__
control_input_ops)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1662, in _create_c_op
raise ValueError(str(e))
ValueError: Cannot reshape a tensor with 172800 elements to shape [16,6,4,30,30] (345600 elements) for 'reshape_1_1/Reshape' (op: 'Reshape') with input shapes: [16,12,30,30], [5] and with input tensors computed as partial shapes: input[1] = [16,6,4,30,30].
Please help me out.