Error when retraining dashcamnet

Hi All,

I followed the tutorial about retraining model, downloaded the dashcamnet model, prepared my tfrecords, and started the tlt-train command. But I got error saying “ValueError: Cannot reshape a tensor with 119808 elements to shape”

I intend to retrain the model to detect “holes” on the road without losing the ability of detecting cars, people, and roadsigns at the same time.

The log is as below:

Blockquote
“/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py”, line 1662, in _create_c_op
raise ValueError(str(e))
ValueError: Cannot reshape a tensor with 119808 elements to shape [4,7,4,34,60] (228480 elements) for ‘reshape_1_1/Reshape’ (op: ‘Reshape’) with input shapes: [4,16,24,78], [5] and with input tensors computed as partial shapes: input[1] = [4,7,4,34,60].

Blockquote

Looks like it is a tensorflow related problem, but where to figure and fix it?

Thanks,

Could you please share your training spec?

detectnet_v2_retrain_resnet18_kitti.txt (9.2 KB)

here is my spec file, Thanks

BTW in the spec, I added several classes which are D00, D10… which stand for different types of holes on road.

In the spec, I did nothing regarding the original detectable objects types that are “cars”,“people”,“roadsigns” and “two-wheelers” cause I think I only intend to add the ability of detecting the “holes”, all my spec should be new detection class related. (which I don’t know if correct or not). If I am wrong, what should I do to include the old classes

thanks,

  1. What is your dataset’s resolution? I am afraid it is 1248x384, right?
  2. Your setting in training spec does not meet requirement. See https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.html#requirements

DetectNet_v2

Input size: C * W * H (where C = 1 or 3, W > =480, H >=272 and W, H are multiples of 16)
Image format: JPG, JPEG, PNG
Label format: KITTI detection

Note: The tlt-train tool does not support training on images of multiple resolutions, or resizing images during training. All of the images must be resized offline to the final training size and the corresponding bounding boxes must be scaled accordingly.

  1. For training, please remove “load_graph: true”
  2. Please refer to Tensor reshape error when evaluating a Detectnet_v2 model too.

My dataset’s images are all jpg format, and 600*600 pixels.
In the spec, I specified
"augmentation_config {
preprocessing {
output_image_width: 600
output_image_height: 600

"

I cannot find other places where I can set the training images size be 600*600.
What should I do with the training images?

Thanks

Please resize images and labels to “W, H are multiples of 16”.

Or you can set
output_image_width: 608
output_image_height: 608

and remove “load_graph: true”