Working through the Mask R-CNN notebook example using two 1080 GPUs on Ubuntu. Reduced batch size to 1. https://developer.nvidia.com/blog/training-instance-segmentation-models-using-maskrcnn-on-the-transfer-learning-toolkit/ At the end of this post is the error message I get during training that t…

According to the log, there is OOM when you train with two 1080 cards. Suggest trying: reduce the bs: try train_batch_size: 1 or train a smaller network. Note that please set width/height to multiples of 64. For example, set image_size: “(640, 1024)” if possible, try other cards, for example,…

Mask R-CNN Training Jupyter Notebook Model Quality and Multiple GPU changes

Accelerated Computing Intelligent Video Analytics TAO Toolkit

Morganh December 22, 2020, 1:51am 3

If you run with 2gpus, please refer to the spec file in Poor metric results after retraining maskrcnn using TLT notebook - #16 by ghazni

Topic		Replies	Views
TLT train maskrcnn model with Mapillary Vistas Dataset failed on CUDA_ERROR_OUT_OF_MEMORY: out of memory TAO Toolkit cuda	86	3250	September 21, 2021
MaskRCNN Input to reshape is a tensor with 3135248 values, but the requested shape has 2691200 TAO Toolkit	38	1211	May 9, 2023
Poor metric results after retraining maskrcnn using TLT notebook TAO Toolkit	23	2490	August 3, 2021
Train mask-rcnn failure TAO Toolkit tao	16	1201	November 25, 2021
Mask R-CNN hangs during training using custom made tfrecords TAO Toolkit	31	3524	October 12, 2021
Input to reshape is a tensor with 3067968 values, but the requested shape has 2691200 TAO Toolkit inception	2	27	January 16, 2025
Cannot reshape a tensor with 25690112 elements to shape [256,256,14,14] TAO Toolkit	51	1425	July 26, 2022
ValueError: Total size of new array must be unchanged for box_head/class-predict/kernel lh_shape: [(1024, 1)], rh_shape: [(1024, 2)] TAO Toolkit	7	971	October 12, 2021
Mask R-CNN hangs during training using custom made tfrecords TensorRT	3	464	April 13, 2021
Mask_rcnn shows training logs loss 0.00000 fast_rcnn class loss: 0.00000 fast_rcnn box loss: 0.00000 TAO Toolkit	22	824	August 16, 2022

Mask R-CNN Training Jupyter Notebook Model Quality and Multiple GPU changes

Related topics