Cannot reshape a tensor with 25690112 elements to shape [256,256,14,14]

Accelerated Computing Intelligent Video Analytics TAO Toolkit
#22

So, can you try 3gpus?

#23

3 GPus also worked.

#24

Please check nvidia-smi again.

#25

nvidia-smi
nvidia-smi976×1000 33.6 KB

#26

Can you kill the job and run 4gpus again?
$ sudo kill -9 173358 173519 173520 173521

#27

still same error

#28

Can you try below experiments?
case1
# mask_rcnn train -e /workspace/tao-experiments/mask_rcnn/specs/maskrcnn_train_resnet18.txt -d /workspace/tao-experiments/mask_rcnn/experiment_dir_unpruned_3gpu -k nvidia_tlt --gpus 3
--gpu_index 0 1 2

case2
# mask_rcnn train -e /workspace/tao-experiments/mask_rcnn/specs/maskrcnn_train_resnet18.txt -d /workspace/tao-experiments/mask_rcnn/experiment_dir_unpruned_3gpu -k nvidia_tlt --gpus 3
--gpu_index 0 1 3

case3
# mask_rcnn train -e /workspace/tao-experiments/mask_rcnn/specs/maskrcnn_train_resnet18.txt -d /workspace/tao-experiments/mask_rcnn/experiment_dir_unpruned_3gpu -k nvidia_tlt --gpus 3
--gpu_index 1 2 3

#29

All worked.

#30

How about
mask_rcnn train -e /workspace/tao-experiments/mask_rcnn/specs/maskrcnn_train_resnet18.txt -d /workspace/tao-experiments/mask_rcnn/experiment_dir_unpruned_4gpu_new -k nvidia_tlt --gpus 4
-- gpu_index 1 2 3 4

#31

Still same errors.
Is it memory issue?

#32

You can set to bs to 1 and retry.
train_batch_size: 1
eval_batch_size: 1

or use less tfrecords.
training_file_pattern: “/workspace/tao-experiments/data/mask_rcnn/data/train*.tfrecord”
validation_file_pattern: “/workspace/tao-experiments/data/mask_rcnn/data/val*.tfrecord”

#33

Yes it works with

train_batch_size: 1
eval_batch_size: 1

for 4 GPUs.

#34

It is stranged, I used to train with 4 GPUs with

train_batch_size: 4
eval_batch_size: 4

for same dataset.
Now can’t

#35

Try a lower num_examples_per_epoch.

#36

That could be the reason. Let me try.

#37

I have 18,000 training images.
What could be appropriate size to set at num_examples_per_epoch?
I like to train on 4 GPUs

#38

Suggest you to open a new terminal to monitor the CPU memory.
Firstly, reproduce above error when run with 4gpus.
Then, decrease num_examples_per_epoch until the issue is not reproduced.

#39

ok thanks

#40

It doesn’t affect my number of images used for training, right?
More epochs will be trained for fewer number of num_examples_per_epoch

#41

The num_examples_per_epoch should be the total number of images in the training set divided by the number of GPUs.