I am attempting to train a new model with my sample data to get a better understanding of how to use TLT.
I am getting the following error when I run the train command:
Total params: 42,937,262
Trainable params: 42,506,158
Non-trainable params: 431,104
__________________________________________________________________________________________________
2019-10-25 19:00:32,725 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loading pretrained weights from /workspace/tlt-experiments/pretrained_models/tlt_resnet50_faster_rcnn_v1/resnet50.h5
2019-10-25 19:00:34,734 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Pretrained weights loaded!
2019-10-25 19:00:34,965 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: training example num: 138
2019-10-25 19:00:35,181 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Starting training
2019-10-25 19:00:35,181 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Epoch 1/12
Found 138 examples in training dataset, valid image extension isjpg, jpeg and png(case sensitive)
Compressed_class_mapping: {u'pore': 0}
Name mapping:{u'pore': u'pore'}
Training dataset stats(compressed via class mapping):
{u'pore': 202}
Traceback (most recent call last):
File "/usr/local/bin/tlt-train-g1", line 10, in <module>
sys.exit(main())
File "./common/magnet_train.py", line 30, in main
File "./faster_rcnn/scripts/train.py", line 273, in main
File "./faster_rcnn/data_loader/loader.py", line 100, in kitti_data_gen
AssertionError: Class pore 0.00 unrecognized in /workspace/tlt-experiments/data/cam/training_labels/Image__2019-07-25__09-09-04.txt
Hi Martin,
There is an additional 0.0 in your pole label text file. Please remove it and try again.
The sum of total number of elements per object is 15. See 4.2.2 in tlt doc for more details.
I now cannot train because I run into the following issue. Please advise me on what may be the problem. (please note, the config.txt remains the same as above.)
2019-10-28 12:27:29,722 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loading pretrained weights from /workspace/tlt-experiments/pretrained_models/tlt_resnet50_faster_rcnn_v1/resnet50.h5
2019-10-28 12:27:31,766 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Pretrained weights loaded!
2019-10-28 12:27:32,001 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: training example num: 139
2019-10-28 12:27:32,248 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Starting training
2019-10-28 12:27:32,248 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Epoch 1/12
Found 139 examples in training dataset, valid image extension isjpg, jpeg and png(case sensitive)
Compressed_class_mapping: {u'pore': 0}
Name mapping:{u'pore': u'pore'}
Training dataset stats(compressed via class mapping):
{u'pore': 203}
No positive ROIs.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-10-28 12:27:41,148 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Traceback (most recent call last):
File "/usr/local/bin/tlt-train-g1", line 10, in <module>
sys.exit(main())
File "./common/magnet_train.py", line 30, in main
File "./faster_rcnn/scripts/train.py", line 309, in main
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1216, in train_on_batch
self._make_train_function()
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 509, in _make_train_function
loss=self.total_loss)
File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/optimizers.py", line 505, in get_updates
self.updates.append(K.update(m, m_t))
File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 973, in update
return tf.assign(x, new_x)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/state_ops.py", line 224, in assign
return ref.assign(value, name=name)
AttributeError: 'Tensor' object has no attribute 'assign'
I now cannot train because I run into the following issue. Please advise me on what may be the problem. (please note, the config.txt remains the same as above.)
2019-10-28 12:27:29,722 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loading pretrained weights from /workspace/tlt-experiments/pretrained_models/tlt_resnet50_faster_rcnn_v1/resnet50.h5
2019-10-28 12:27:31,766 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Pretrained weights loaded!
2019-10-28 12:27:32,001 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: training example num: 139
2019-10-28 12:27:32,248 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Starting training
2019-10-28 12:27:32,248 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Epoch 1/12
Found 139 examples in training dataset, valid image extension isjpg, jpeg and png(case sensitive)
Compressed_class_mapping: {u'pore': 0}
Name mapping:{u'pore': u'pore'}
Training dataset stats(compressed via class mapping):
{u'pore': 203}
No positive ROIs.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-10-28 12:27:41,148 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Traceback (most recent call last):
File "/usr/local/bin/tlt-train-g1", line 10, in <module>
sys.exit(main())
File "./common/magnet_train.py", line 30, in main
File "./faster_rcnn/scripts/train.py", line 309, in main
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1216, in train_on_batch
self._make_train_function()
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 509, in _make_train_function
loss=self.total_loss)
File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/optimizers.py", line 505, in get_updates
self.updates.append(K.update(m, m_t))
File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 973, in update
return tf.assign(x, new_x)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/state_ops.py", line 224, in assign
return ref.assign(value, name=name)
AttributeError: 'Tensor' object has no attribute 'assign'